When to call an A/B test: significance, sample size and the peeking problem
George Peric · CRO designer · 7 min readShort answer
Call an A/B test when it has reached the sample size and duration you planned before launch, counted from the last change to the test, and the result is statistically significant on your primary metric with no guardrail problems. Stopping the moment you first see p<0.05 inflates false positives. Plan the sample, run full weeks, then read the result once.
Key takeaways
- Checking results repeatedly and stopping at the first significant reading makes false winners much more likely.
- Decide the sample size, the minimum effect you care about and the duration before launch. Run full weeks.
- Count results from the last change to the test: the final ramp step, a variant fix or a tracking fix.
- Sequential and Bayesian methods allow earlier decisions, but only when the tool is built for it and you follow its rules.
- A flat result is an answer too. Decide in advance what you will do if the test is inconclusive.
Why shouldn’t you stop an A/B test as soon as it is significant?
Because significance tests assume you look once, at the end, and every extra look gives chance another opportunity to cross the line. Early in a test, results swing a lot. If you check every day and stop the first time p drops below 0.05, you will often stop on a swing rather than a real difference. Evan Miller’s well-known article on this puts it plainly: repeated significance testing always increases the rate of false positives.
The fix is not to stop looking at the test. Look at it every day for bugs, sample ratio mismatch and guardrails. Just don’t use the significance reading to decide when to stop. As Conversion.com puts it, significance should not dictate when you stop a test.
How do you decide sample size and duration before launch?
Pick your primary metric, its current rate, the smallest effect worth detecting, and your significance and power levels, then use a sample size calculator. The calculator gives visitors per version. Divide by the daily visitors each version will get to find the number of days, then round up to full weeks.
- Baseline: the current conversion rate or revenue per visitor of the control.
- Minimum detectable effect: the smallest lift that would be worth acting on. Smaller effects need far more traffic.
- Significance level and power: a common choice is a 5% significance level and 80% power (explained below).
- Full weeks: weekday and weekend visitors often behave differently. Running whole weeks keeps each day of the week equally represented. I don’t run a test for less than two full weeks.
Example
Say a landing page converts at 3% and you want to detect a 10% relative lift (3.0% to 3.3%). Evan Miller’s rule of thumb, n = 16 × p(1 − p) / δ², gives about 16 × 0.0291 / 0.000009, or roughly 52,000 visitors per version. If the page gets 8,000 visitors a day, each version gets 4,000, so the test needs about 13 days. Round up to 14 days: two full weeks.
For more on the inputs and what to do when traffic is low, see A/B test sample size.
What do statistical significance and power mean, in plain English?
Significance controls how often you call a winner that isn’t real. Power controls how often you catch a winner that is real.
- Significance level: the share of the time the test will show a difference when there is none. Evan Miller’s calculator defines it as the percent of the time a difference will be detected, assuming one does not exist. At 5%, about one in twenty tests of a change that does nothing will still look like a winner.
- p-value: how surprising your result would be if the variant had no real effect. A small p-value means “this would be unusual if nothing changed”. It is not the probability that the variant is better.
- Power: the share of the time the test will detect your minimum effect, if that effect really exists. At 80% power, a real lift of that size is missed about one time in five.
- Minimum detectable effect: the smallest real lift the test is sized to find at that power. Lifts smaller than this can easily come out flat.
From what point should you count A/B test results?
From the last change to the test. Anything before that point is a different experiment. Changes that reset the clock:
- A ramp step, if it changed the split between control and variant. Mixing periods with different splits can bias the pooled result. See how to ramp up an A/B test.
- A variant fix, such as a bug fix, a copy change or a layout change on the variant.
- A tracking fix, such as a conversion event that was missing, doubled or sent the wrong value.
From that point, keep going until two things are true: the pre-planned sample size and duration are reached, and the result is significant on the primary metric. If the sample is reached and the result isn’t significant, the test is inconclusive. Don’t keep extending it in search of significance. Optimizely recommends resetting results when moving from a ramp stage to full allocation, and LaunchDarkly prompts users to reset results after a ramp to avoid time-based bias.
Example
Say a test starts on Monday, and on Thursday you find the purchase event wasn’t firing on the variant for Safari users. You fix it Thursday afternoon. The test now starts on Thursday. If the plan was two full weeks, it ends two weeks after that Thursday, and the first three days don’t count toward the sample or the result.
Are sequential testing and Bayesian A/B testing better?
They can let you decide earlier, but they are not a free pass to peek.
- Sequential testing plans the checkpoints in advance, with stricter thresholds at each look so the overall false positive rate stays where you set it. Evan Miller describes it as setting up checkpoints in advance where you decide whether to continue. It works when the tool implements it properly and you stick to its rules.
- Bayesian testing reports things like “probability the variant is better”, which many people find easier to read. It still depends on the prior and the decision rule the tool uses, and stopping the moment a probability looks high can still mislead if you ignore those rules.
Honestly, the method matters less than the discipline. A fixed-sample test read once at the planned end, a proper sequential test and a sensible Bayesian setup can all work. Constant checking of a fixed-sample p-value with no plan does not.
What are guardrail metrics and why do they matter?
Guardrail metrics are the numbers that must not get worse, even if the primary metric improves. A variant can raise form submits while lowering lead quality, or raise orders while lowering order value. Common guardrails:
- Revenue per visitor, if the primary metric is conversion rate.
- Average order value, refund or cancellation rate.
- Lead quality or qualified lead rate.
- Page speed and error rates.
- Sample ratio: the visitor split matches the one you set. Microsoft’s experimentation team advises not trusting results with a sample ratio mismatch until the cause is found.
Decide your guardrails before launch, with a rough line for what counts as harm. A “winner” that breaks a guardrail is not a winner.
What should you do with flat or inconclusive results?
Accept them, write down what you learned, and decide the next step based on the hypothesis, not on a hunch. Flat results are common. Options:
- Keep the control if the variant costs more to maintain and showed no gain.
- Ship the variant if it is better for other reasons (simpler, on-brand, needed for a bigger change) and the guardrails were fine. Be honest that it isn’t a proven lift.
- Test a bolder version. If the change was small, the real effect may be below what the test could detect. A bigger change is more likely to move the numbers.
- Check segments carefully. A segment that “won” in a flat test is a new hypothesis to test, not a result.
To decide what to test next, see how to prioritize A/B test ideas and A/B test ideas for landing pages.
Before you call a winner: what should you check?
Run through this checklist before you report any result. I use a version of it on every test I design.
| Check | What “yes” looks like |
|---|---|
| Plan was set before launch | Primary metric, minimum effect, sample size and duration written down before the test started |
| Counting from the last change | Data starts after the final ramp step, the last variant fix and the last tracking fix |
| Sample size reached | Each version has at least the planned number of visitors since the last change |
| Full weeks | The test ran whole weeks, at least two, since the last change |
| Significant on the primary metric | p-value below your threshold (or the sequential or Bayesian rule met), read at the planned end |
| No sample ratio mismatch | Visitor counts match the split you set |
| Tracking parity | All events fire once on both versions, with the right values |
| Guardrails held | Revenue per visitor, order value, quality and error rates did not get worse |
| No outside disruption | No big sale, outage or campaign change hit one version more than the other |
| Effect size makes sense | The lift is plausible for the change made; a huge lift from a tiny change gets a second look |
Once a winner passes, turn the result into money with how to calculate the revenue impact of an A/B test. If you want help planning and reading tests, see A/B testing or book a call.
FAQ
How long should an A/B test run?
Until it reaches the sample size you planned before launch, in full weeks, and at least two weeks. The number of days comes from your traffic, baseline rate and the smallest effect you want to detect, so use a sample size calculator rather than a fixed rule.
Can I stop an A/B test early if it is clearly winning?
Not with a standard fixed-sample test, because early leads often shrink. If you need the option to stop early, use a sequential testing method that plans the checkpoints in advance. Do stop early if the variant is broken or harming a guardrail.
What does 95% significance mean in A/B testing?
It means a 5% significance level: if the variant truly made no difference, a result this extreme would show up only about 5% of the time. It does not mean there is a 95% chance the variant is better.
What should I do if my A/B test is inconclusive?
Treat it as an answer: the change didn’t make a difference big enough to detect. Keep the control or ship the variant for other reasons, and if you still believe in the idea, test a bolder version of it.
Sources
I’m George (Djordje) Peric, a CRO designer, CXL certified in Conversion Optimization and Ecommerce Marketing. I design ads, landing pages and funnels and A/B test them.