A/B Test Traffic Splitting: Statistical Validity at Low Volume
Summary: On a small site, the binding constraint is not visitor count but the false-positive rate you accept while monitoring. Fix the split first, pre-commit to a detectable effect, then choose a method that tolerates looking.
Key Takeaways
- Peeking turns 5% into as much as 30%. Continuous monitoring with fixed-horizon tests inflates the false positive rate by up to six times.
- 50 visitors per arm is below the floor. Below that, a "winner" is usually the noise in the first two days.
- Set the MDE before traffic, not after. On a small site the realistic detectable lift is 20 to 30% relative, and anything smaller needs a different instrument.
- Split by user, not by session. Session-level splits leak the same person into both arms and quietly cancel the effect.
- Two verdicts, one dataset. A p-value of 0.061 and a 96.9% Bayesian probability can describe the same 8,000-visitor test.
I. The real constraint is the false positive rate
Low-traffic teams usually frame the problem as "we do not have enough visitors." That framing leads to the wrong fix — longer tests, or worse, waiting for a p-value that never arrives. The constraint that actually binds is the error rate you are willing to run while the test is live.
With a fixed-horizon test, the 5% alpha is only honest if you look exactly once, at the sample size you planned. Check the dashboard every morning and stop when it looks good, and the effective false positive rate climbs toward 30%. Research on continuous monitoring in this setting found exactly that: nominal 5% becomes as high as 30% when people peek daily and stop on significance. Nearly one in three "winners" is then a false positive.
For a site doing 200 conversions a month, that arithmetic is fatal. Three false winners a quarter means the roadmap is being steered by noise, and the loss is not statistical — it is the quarter spent building the wrong thing, plus the compounding loss of not having shipped real improvements.
There is a second-order effect that teams rarely price in: a false winner burns more than its own development time. It becomes the new baseline, so the next test inherits a change that never worked, and the stacked loss compounds quietly across a quarter. That is why the error rate matters more than the sample size — one bad decision contaminates the sequence, while a slow but honest test only costs time.
II. Traffic splitting is where validity is won or lost
Before any statistical choice, the assignment mechanism has to be sound. Four failure modes account for most invalid low-traffic tests:
| Split design | What breaks | Fix |
|---|---|---|
| Session-level randomisation | Same visitor flips arms across devices | Hash on user or account id |
| Time-based sequential (A two weeks, then B) | Seasonality, promotions, payday cycles | Run parallel, or block by day |
| Unbalanced buckets from sampling | 52/48 split read as a real lift | Verify split ratio before reading results |
| Logged-in users excluded after the fact | Composition change mid-test | Freeze the analysis population at launch |
The session-level one is the most common and the least discussed. If a visitor sees variant A on mobile and variant B on desktop, their behaviour averages the two experiences, and the measured lift shrinks toward zero. Teams then conclude "the change did not work," when the experiment never isolated it.
A cheap pre-flight check: before launching, log the assignment for 1,000 visitors and verify the ratio is within about two percentage points of the intended split, and verify that each id appears exactly once. Both bugs — imbalance and duplicate assignment — are diagnosable in an hour and invisible after launch.
III. The MDE budget is arithmetic, not opinion
Minimum detectable effect is the smallest lift your test can resolve. It is set by traffic, baseline rate and the alpha/power you accept, and it should be computed before the test, not discovered after it.
Worked example: a page with 4,000 monthly visitors and a 4% conversion rate yields 160 conversions per month. Splitting evenly gives 80 conversions per arm per month. At conventional 80% power and 5% alpha, detecting a 20% relative lift — 4% to 4.8% — requires roughly 8,000 visitors per arm, which is about 100 weeks on this traffic. That number is the answer to "why does our test never finish," and it is knowable in five minutes with a sample-size calculator.
Two honest responses follow. Either raise the effect size you commit to testing — a redesign or a pricing change, not a button colour — or stop calling it an experiment and ship on judgement with a before/after read. Sending a 10% relative lift target to a 4,000-visitor page is not a test; it is a queue.
A practical rule for small sites: only test changes you expect to move conversion by 20 to 30% relative. Below that, use a different instrument — session recordings, funnel drop-off analysis, or a qualitative read — because the test cannot see the effect you are chasing.
One more input is usually missing when the budget is computed: revenue per visitor rather than conversion rate. Two variants with identical conversion rates can differ 20% in revenue if order value shifts, and a test powered for conversion rate alone will miss that. On small sites, revenue per visitor is the better primary metric precisely because it folds average order value into one number and needs fewer visitors to separate the arms.
IV. Choosing a method that survives monitoring
If you will look at the dashboard daily — and you will — the method has to tolerate looking. The honest options:
- Anytime-valid confidence sequences. Sequential methods built so that continuous monitoring does not inflate error. Adobe deployed exactly this on its experimentation platform, and reports type-I error staying below the 5% threshold under continuous monitoring where a peeked fixed-horizon test drifts toward 1.
- Bayesian with a pre-registered decision rule. Stop when the probability to be best crosses a threshold, with a hard floor on sample per arm — 50 visitors is an absolute minimum, not a target. The posterior updates continuously and does not punish peeking the way a p-value does.
- Group sequential designs. Pre-plan a small number of interim looks with an alpha-spending function. Fewer looks, but each is legitimate.
The same dataset can yield opposite verdicts, and that is worth seeing once. A test running to 8,000 visitors per arm closed at 320 conversions in control and 368 in the variant — a 15% relative lift. The frequentist read returned a p-value of 0.061 and a confidence interval crossing zero: not significant. The Bayesian read with a uniform prior returned a 96.9% probability that the variant was better. Neither is wrong; they answer different questions, and the decision rule has to be chosen in advance so that the number does not pick the story after the fact.
That is the discipline that matters more than the framework. Write the stopping rule, the MDE and the primary metric into the test document before launch, then let the pre-registered rule decide. A team that switches frameworks after seeing the data has not bought statistical rigour; it has bought a rationalisation.
A note on duration, because "run it longer" is the default advice and it is not always right. Time confounders accumulate with duration: a two-week test can straddle a payday and a campaign, and the longer it runs the more likely that composition shifts because a new traffic source started. Six weeks does not quadruple certainty if the audience changed halfway through; it just adds newer, different visitors to a comparison that was designed around the old ones.
V. Boundaries: when the test should not run
Three situations where the statistically correct answer is "do not test":
- Under roughly 5,000 monthly visitors to the test page, most conversion tests cannot resolve anything worth shipping. The exceptions are large structural changes where a 30% relative effect is plausible.
- If the baseline rate is under 1%, fix the funnel first. At that level the variance is dominated by a handful of edge cases — payment failures, broken mobile flows — and those are found by reading logs, not by randomising.
- If you cannot hold the rest of the site still, the test measures the noise you introduced. A simultaneous pricing change or a campaign launch makes the arms incomparable, and no method repairs that.
One more boundary worth stating: statistical significance and practical significance are different questions. A 2% relative lift on a page that produces 40 conversions a month is detectable with enough patience and worth roughly nothing. Before launching, ask what annual revenue the smallest interesting effect implies — for many small sites the answer is "less than the cost of the test," and that is a legitimate reason to skip it.
FAQ
Q: How many visitors do we need per variant?
A: 50 is a floor, not a target. A 4% baseline with a 20% lift target needs about 8,000 per arm.
Q: Why is daily checking a problem?
A: With fixed-horizon tests it inflates the false positive rate from 5% toward 30%; use sequential or Bayesian methods instead.
Q: Should we split by session or by user?
A: By user or account. Session splits leak the same person into both arms and shrink the measured effect.
Q: What if the Bayesian and frequentist verdicts disagree?
A: Pick the rule before launch. On the same 8,000-visitor test they can say 0.061 and 96.9% at once.
Q: Is a 2% lift worth testing?
A: Usually not on low traffic. Test changes expected to move 20 to 30% relative, or use funnel analysis instead.
About Tengence
Tengence Insight segments traffic by user identity, holds the analysis population fixed at launch, and reports lift with the uncertainty attached, so a small site can tell a real effect from a Monday.
Related Reading
Data Sources
- Statistical significance in A/B testing: when to call it — continuous monitoring of fixed-horizon tests inflates the false positive rate from the nominal 5% to as high as 30%; MDE is the most overlooked input; significance and practical significance are separate questions.
- A/B testing for small eCommerce stores — below roughly 5,000 monthly visitors to the test page, results take months and are unreliable; set MDE at 20 to 30% relative lift; prefer parallel over sequential to avoid seasonality confounds; peeking inflates error rates.
- Same dataset, two verdicts — 8,000 visitors per arm, 320 versus 368 conversions, 15% relative lift; frequentist p-value 0.061 with a confidence interval crossing zero; Bayesian probability to be best 96.9%.
