How many visitors do you need for an A/B test?
George Peric · CRO designer · 7 min readShort answer
It depends on your baseline conversion rate and the smallest lift you want to detect. At a 5% conversion rate, detecting a 10% relative lift (5% to 5.5%) at 95% significance and 80% power needs about 31,200 visitors per variant, or about 62,500 for a two-variant test. Lower conversion rates and smaller lifts need far more. Turn that number into full weeks of traffic, and run for at least two weeks.
Key takeaways
- Five inputs decide sample size: baseline conversion rate, minimum detectable effect, significance level, power and number of variants.
- Halving the lift you want to detect roughly quadruples the visitors you need.
- Divide the total sample by daily traffic, round up to full weeks, and never run less than about two weeks.
- Low traffic? Test bigger changes, on higher-traffic pages, with fewer variants or an earlier funnel metric.
What decides how many visitors an A/B test needs?
Five inputs: your baseline conversion rate, the minimum detectable effect, the significance level, the statistical power and the number of variants.
- Baseline conversion rate: how often the current page converts today, for the same traffic that will be in the test. Lower rates need more visitors.
- Minimum detectable effect (MDE): the smallest lift you want the test to be able to catch. Smaller effects need many more visitors.
- Significance level (alpha): the false positive rate you accept. The usual choice is 0.05, which is the “95% significance” most tools show.
- Power: the chance the test catches a real effect of at least the MDE. The usual choice is 80%.
- Number of variants: every variant needs its own full sample, so each extra variant adds to the total.
What is the difference between relative and absolute MDE?
A relative MDE is a percentage of the baseline, an absolute MDE is a change in percentage points. On a 5% baseline, a 10% relative lift means 5% to 5.5%. A 1 point absolute lift means 5% to 6%, which is a 20% relative lift. Mixing the two up is one of the most common reasons sample size estimates are off by a factor of four. Calculators differ here. Evan Miller’s calculator lets you choose either, so check which one your tool expects before you type a number in.
What is the sample size formula for an A/B test?
For a conversion rate, the standard approximate formula per variant is:
n = (zα/2 + zβ)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)²
- p₁ is the baseline conversion rate and p₂ is the rate you want to be able to detect.
- zα/2 is 1.96 for alpha 0.05, two-sided.
- zβ is 0.84 for 80% power.
- So (zα/2 + zβ)² is about 7.85 for the usual settings.
In plain English: the noisier the metric (the p(1 − p) part) and the smaller the difference you want to see (the bottom of the fraction), the more visitors you need. Because the difference is squared, halving the MDE roughly quadruples the sample.
Example
Say a landing page converts at 5% and you want to detect a 10% relative lift, so p₁ = 0.05 and p₂ = 0.055.
- p₁(1 − p₁) = 0.05 × 0.95 = 0.0475
- p₂(1 − p₂) = 0.055 × 0.945 = 0.051975
- Sum = 0.099475
- (p₂ − p₁)² = 0.005² = 0.000025
- 0.099475 ÷ 0.000025 = 3,979
- 3,979 × 7.85 = about 31,200 visitors per variant
With a control and one variant, that is about 62,500 visitors in total.
Different calculators use slightly different versions of this formula, so the same inputs can give somewhat different answers from tool to tool. Pick one calculator and use it for every test so your plans are comparable.
How many visitors per variant for common conversion rates?
Here are the visitors needed per variant at alpha 0.05 (two-sided) and 80% power, computed with the formula above. Multiply by the number of variants, control included, for the total.
| Baseline conversion rate | MDE 5% relative | MDE 10% relative | MDE 20% relative |
|---|---|---|---|
| 2% | 315,204 (2% to 2.1%) | 80,679 (2% to 2.2%) | 21,106 (2% to 2.4%) |
| 5% | 122,121 (5% to 5.25%) | 31,231 (5% to 5.5%) | 8,155 (5% to 6%) |
| 10% | 57,760 (10% to 10.5%) | 14,749 (10% to 11%) | 3,839 (10% to 12%) |
Two patterns stand out. Going from a 20% to a 5% MDE multiplies the sample by roughly 15. And a 2% baseline needs about five and a half times the traffic of a 10% baseline for the same relative lift.
How do you turn sample size into test duration?
Divide the total sample by the daily traffic that enters the test, round up to whole weeks, and run at least two full weeks.
- Total sample = visitors per variant × number of variants.
- Days = total sample ÷ daily visitors in the test (only the traffic that actually sees the tested page).
- Round up to full weeks, so every day of the week is counted equally. Weekday and weekend visitors often behave differently.
- Run at least about two weeks, even if the math says less, so one odd day or campaign doesn’t decide the result. Optimizely’s own guidance is at least one business cycle of seven days. I prefer two.
Example
Say the 5% page above gets 5,000 visitors a day. A two-variant test needs about 62,500 visitors, which is 12.5 days. Rounded up to full weeks, that is a 14-day test. Add a third variant and the total becomes about 93,700 visitors, or 19 days, which rounds up to 21 days.
If you ramp traffic up slowly at the start, count only the days at full allocation. See how to ramp up an A/B test and when to read A/B test results for why stopping early on a “significant” day is a mistake.
What can you do with low traffic?
Make the effect bigger or the traffic bigger, since those are the two levers that move the sample the most.
- Test bigger changes. A new offer, a new first screen or a full redesign can plausibly move results by 20% or more. A button color won’t, and a test sized for 5% needs about 15 times the visitors.
- Test on higher-traffic pages. The landing page that gets all the paid traffic beats a deep page that few people reach.
- Use an earlier funnel metric. Clicks to the next step or form starts happen more often than purchases, so they need fewer visitors. Keep revenue as a guardrail so you don’t win on clicks and lose on money.
- Use fewer variants. One variant against the control. Each extra variant adds a full sample and, if you correct for multiple comparisons, raises the sample per variant too.
- Accept a larger MDE on purpose. If the page can only detect a 20% lift in a reasonable time, only test ideas that could deliver that.
Traffic is also part of prioritization. In an ICE score sheet, an idea on a page that can’t reach a result within a few weeks should get a lower Impact score.
Should you count sessions or users?
Count the unit your testing tool randomizes on, which is usually the user (visitor), not the session. Most tools assign a visitor to a variant with a cookie and keep them there across visits. Several sessions from the same person are not independent, so counting sessions overstates your sample. Make the baseline conversion rate use the same unit too: conversions per user, if the tool splits by user.
Why do revenue metrics need more data?
Because revenue per visitor is much noisier than a conversion rate. Most visitors spend nothing, and the ones who buy spend very different amounts, so a few large orders can swing the average. The conversion rate formula above doesn’t apply. For a metric like revenue per visitor, the per-variant formula uses the standard deviation (σ) of the metric instead:
n = 7.85 × 2σ² ÷ δ², where δ is the smallest difference in revenue per visitor you want to detect.
Take σ from your historical data for the same page and traffic. Expect the answer to be larger than for a conversion rate on the same page. That is still the metric I judge tests on, because it captures both conversion and order value. See how to calculate the revenue impact of an A/B test for how to turn the result into yearly money.
In six years of designing for A/B tests, the most common planning mistake I have seen is starting a test the page can’t finish. Do the math before the design work starts. If you want help sizing and running tests, see A/B testing or book a call.
FAQ
What is a good sample size for an A/B test?
There is no single number. It depends on your baseline conversion rate and the smallest lift you want to detect. At a 5% conversion rate and a 10% relative lift, you need about 31,200 visitors per variant at 95% significance and 80% power.
Can I stop the test early if it reaches significance?
Not if you planned a fixed sample size. Checking repeatedly and stopping on the first significant day raises the false positive rate. Run to the planned sample and full weeks, unless your tool uses a sequential method designed for early stopping.
What is minimum detectable effect?
It is the smallest lift your test is designed to catch reliably. If the real effect is smaller than the MDE, the test will often miss it. Check whether your calculator expects it as a relative or an absolute change.
Why should a test run at least two weeks?
Visitors behave differently on different days of the week, and campaigns or emails can make one day unusual. Two full weeks cover each weekday twice and make the result less likely to depend on one odd day.
Sources
I’m George (Djordje) Peric, a CRO designer, CXL certified in Conversion Optimization and Ecommerce Marketing. I design ads, landing pages and funnels and A/B test them.