The Complete Guide to A/B Testing: Sample Size, Duration, and Significance

Summary: The credibility of an A/B test rests on 3 things: collecting enough sample, running a full business cycle, and judging significance against a stopping rule fixed in advance. This guide connects sample size calculation, duration setting, significance testing, multiple-comparison correction and null-result handling into one executable loop, and adds workable alternatives for low-traffic scenarios.

Most teams do not fail at A/B testing because their ideas are bad. They fail because the conclusion is not trustworthy: calling a result before the sample is complete, stopping early when day-one data looks good, or tracking 5 metrics and reporting only the one that improved. All of these push the false positive rate from a nominal 5% up to 20% or higher. This article follows the order "sample size, duration, significance, multiple testing, wrap-up" and gives the concrete calculation and judgement rule for each step, so experimental results can genuinely be used for decisions.

Key Takeaways

Sample size is decided jointly by baseline conversion rate, minimum detectable effect (MDE) and statistical power; if any of the three is missing, it cannot be calculated.
Every time the MDE is halved, the required sample roughly quadruples; low-baseline pages need a bigger change, not a longer wait.
Test duration must satisfy both the sample size requirement and a full business cycle; the shortest practical run is usually no less than 14 days.
Checking results repeatedly and stopping early (peeking) pushes the false positive rate from 5% to over 20%; the stopping rule must be fixed in advance.
When you observe several metrics or several variants at once, apply multiple-comparison correction, otherwise "significant" is simply a probabilistic certainty.
Only about 1 in 7 experiments produces a clear winner; null and negative results are the norm and should be recorded and attributed just the same.
Low-traffic pages should not force a classic A/B test; switch to sequential testing, Bayesian methods or qualitative research.

1. Sample Size Is Set by Baseline Conversion, MDE and Power Together

The essence of sample size is this: given a baseline conversion rate, how much sample do you need to reliably tell a real lift apart from random noise. For the most common comparison, a conversion-rate test (two-proportion test), the sample per group can be estimated with n ≈ 2 × (Z_{1-α/2} + Z_{1-β})² × p × (1-p) / δ², where p is the baseline conversion rate, δ is the absolute lift (the MDE expressed in percentage points), α is the significance level, and 1-β is statistical power. Using the industry-standard 95% confidence level and 80% power, with a baseline conversion rate of 5% and a desired absolute lift of 1 percentage point, each group needs about 7,415 visitors. This parameter set makes one key fact clear: sample size is not a round number pulled out of thin air, it is the result of solving for 4 variables at once.

Setting the MDE is the step most likely to go wrong. The smaller the MDE, the weaker the signal you need to distinguish, so the larger the sample required, and the relationship is quadratic: halve the MDE and the sample roughly quadruples. On a page with a 2.5% baseline, detecting a 10% relative lift (that is, 0.25 percentage points) needs about 15,700 visitors per group; raise the expectation to a 20% relative lift and the sample drops to roughly a quarter. Pages with a low baseline and low traffic therefore should not obsess over detecting tiny changes. Design the experiment around a difference large enough that limited traffic can still produce an answer.

2. Turn Sample Size Into an Executable Target With a Benchmark Table

Once you have the sample size, the next step is converting it into a target the team can understand. A benchmark table laid out by "baseline conversion rate × relative lift" is the most practical tool: put 3 baseline tiers, 2.5%, 5% and 10%, on the horizontal axis, 2 relative-lift tiers, 10% and 20%, on the vertical axis, and fill each cell with the required sample per group plus the corresponding total sample (number of groups × sample per group). With this table in hand, a product manager can judge at requirement time whether a change is detectable at current traffic, instead of discovering insufficient power only after the experiment has finished.

One point deserves emphasis: an underpowered experiment is more dangerous than no experiment at all. It ends with a "no significant difference" conclusion, the team concludes the change does not work and drops it, when in reality the sample was simply too small. Before launch, make "how many days to reach 80% power at current traffic" a hard gate for experiment admission. If the time required exceeds the acceptable window (usually more than 4 weeks), redesign the experiment: raise the MDE, cut the number of variants, or move the test to a higher-traffic page.

3. Duration Must Satisfy Sample Size and a Full Business Cycle

Duration is not a simple conversion of sample size; it also has to cover the full rhythm of the business. Purchase behaviour varies clearly across the week: the traffic mix, decision time and average order value on weekdays can all differ from weekends, so running only 5 working days biases the result toward weekday-type users. The general rule is to run at least 2 complete business cycles (that is, 14 days), and not longer than 8 weeks: beyond 8 weeks an experiment faces shifting traffic composition, seasonality and interference from external events, and historical data becomes less representative of new data.

2 floors are also worth memorising: at least 1,000 sessions per group and at least 30 conversions per group. The first keeps the traffic sample from being too thin; the second gives the conversion-rate estimate itself a basic level of stability. If the experiment has not reached its sample size after 14 days, keep running until it does rather than stopping by the calendar; conversely, if the sample is reached within 3 days, you still have to run at least one complete business cycle. Duration and sample size are joined by "and", not "or".

4. Significance Judgement Means Refusing to Peek and Fixing the Stopping Rule

Statistical significance is widely misused, and the classic error is peeking: checking results repeatedly while the test runs and stopping the moment the p-value drops below 0.05 to declare victory. This turns a single test into many tests, and the false positive rate accumulates with them: a nominal 5% significance level can climb past 20% under repeated inspection; checking once a day with a readiness to stop at any time can push it beyond 30%; in the extreme case of checking after every visitor, the false positive rate can even reach the neighbourhood of 26%.

The correct approach is to fix the stopping rule in advance: before the experiment starts, write down "test once when N samples have accumulated", and in between look only at health-check metrics (whether traffic splitting is balanced, whether there are technical errors, whether extreme outliers appear), never at the significance of the primary metric. If stopping mid-flight is genuinely necessary, use only statistical methods designed for sequential settings, such as the sequential probability ratio test or Bayesian stopping rules, rather than repeatedly executing a fixed-sample test. In addition, the p-value only answers whether the difference could have arisen by chance; it does not answer how large the difference is, so confidence intervals and the relative lift must always be reported alongside it.

5. Multiple Metrics and Multiple Variants Need Multiple-Comparison Handling

An experiment that observes only one metric is rare inside an enterprise. The common practice is to watch click-through rate, add-to-cart rate, conversion rate and average order value at the same time, or to launch 3 or 4 variants simultaneously. The problem is that every extra test is another chance to hit significance at random. When you observe 10 mutually independent metrics, even if the change is completely ineffective, you should expect about 0.5 of them to reach 5% significance; if "any metric significant" counts as success, the overall false positive rate sits far above 5%.

There are 3 layers of response. The first is to separate primary from secondary: designate a single primary metric in advance (guardrail metrics are only checked for "must not degrade"), and count the experiment as successful only when the primary metric passes. The second layer is statistical correction: the simplest is the Bonferroni correction, dividing α by the number of tests (with 5 tests, α becomes 0.01); you can also control the false discovery rate (FDR) to trade a little rigour for sensitivity in exploratory work. The third layer is to treat multi-variant cases separately: the more variants, the worse the multiple-comparison problem, and the less traffic each group receives, so the usual recommendation is no more than 4 variants per experiment, including the control.

6. Null and Negative Results Are Conclusions Too

Long-running industry statistics show that only about 1 in 7 experiments produces a clear winner; the rest are mostly null or negative results. That is not a failure of the method, it is the true distribution of product optimisation: changes that genuinely and significantly shift user behaviour are simply rare. Teams should therefore not treat "produced a significant lift" as the only definition of a successful experiment. Ruling out a wrong direction at acceptable cost counts as valid output too.

The right way to handle a null result is attribution, not amnesia: was the MDE set too small, leaving the test underpowered? Was the change so minor that users never noticed it? Or did the change indeed touch an irrelevant part of the journey? Recording the cause of every null result markedly improves the design quality of later experiments. Negative results deserve even more attention: they prove definitively that a direction does not work, and often carry more information than a positive result. When an experiment ends, regardless of the conclusion, archive the design, the sample-size rationale, the full metrics (including confidence intervals) and the conclusion, and build a searchable experiment library.

FAQ

Q: How much sample does an A/B test actually need?

There is no fixed number; it depends on 4 parameters: baseline conversion rate, MDE, significance level and statistical power. At 95% confidence and 80% power, a 5% baseline with a 1 percentage point absolute lift needs about 7,415 visitors per group; drop the baseline to 2.5% and detect a 10% relative lift and you need about 15,700 per group.

Q: How should the MDE be set?

The MDE should equal the smallest lift that makes the change worth shipping, usually derived backwards from business value and implementation cost, and generally set at a 10%–20% relative lift. Note that halving the MDE roughly quadruples the sample, so pages with limited traffic should set a larger MDE.

Q: How many days does an experiment have to run?

At least 2 complete business cycles (usually 14 days) to cover within-week variation, and no more than 8 weeks. At the same time you must satisfy the floors of at least 1,000 sessions and 30 conversions per group. Days and sample size are joined by "and": only when both are met can you stop.

Q: Why can't I look at results mid-flight and stop when it turns significant?

Checking repeatedly and stopping whenever you like turns one test into many, and the false positive rate accumulates from a nominal 5% to over 20%, or beyond 30% if you check daily. Fix the inspection point in advance, and in between look only at technical health metrics such as traffic-split balance.

Q: Is watching several metrics at once a problem?

Yes. The more metrics, the more chances to hit significance at random. Designate a single primary metric in advance and treat the others as guardrails checked only for degradation; if judgement across multiple metrics is truly needed, use the Bonferroni correction (with 5 tests, α becomes 0.01) or control the false discovery rate.

Q: What should I do when the experiment returns "no significant difference"?

First judge whether power was insufficient: work backwards from actual traffic to the smallest MDE you could detect and compare it with the expected lift. If it is larger, the conclusion is "could not detect", not "no effect". If power was sufficient, record the cause (change too light, direction irrelevant, and so on) and archive it to avoid repeating the same attempt.

Q: How do you run experiments on a page with very little traffic?

3 routes. First, switch to sequential testing or Bayesian methods, which allow early stopping once enough evidence has accumulated. Second, give up on tiny changes and design versions different enough to match the traffic you have. Third, replace quantitative testing with qualitative methods such as user interviews and usability testing: find the direction first, then validate it quantitatively.

About Tengence

Scientific experiment design and effect measurement are the precondition for building growth decisions on evidence rather than intuition. Tengence focuses on growth technology for public-traffic acquisition and private-domain conversion, and the experiment design and effect measurement covered in this article are among the areas it follows continuously.

Data Sources

PulseCRO: How to Calculate the Right Sample Size for Your A/B Test: gives the two-proportion sample size formula and the meaning of each parameter, and lists the estimate of about 7,415 visitors per group at a 5% baseline, an MDE of 1 percentage point, and 95% significance / 80% power, recommending an MDE of 10%–20% relative lift.
Metricuno: Sample Size in A/B Testing: provides a sample-size benchmark table at 95% confidence / 80% power, notes that halving the MDE roughly quadruples the sample, and gives the example of about 15,700 per group for detecting a 10% relative lift on a 2.5% baseline.
Grow Conversions: How Long to Run an A/B Test: explains that peeking pushes the false positive rate from 5% above 20%, and gives the window guidance of a minimum of 2 complete business cycles (14 days) and a maximum of 8 weeks, plus the floors of 1,000 sessions and 30 conversions per group.
Clarigital: A/B Testing Methodology: gives the finding that multiple interim checks (10 of them) drive the false positive rate above 30%, and the practice of tightening α to 0.01 in the Bonferroni correction.
Systeme.io: A/B Testing: gives the long-run statistic that about 1/7 of experiments produce a clear winner, and the estimate that peeking after every visitor raises the false positive rate to about 26%.