How to ramp up an A/B test (and why not to start at 50/50)
George Peric · CRO designer · 8 min readShort answer
Ramping up an A/B test means sending a small share of traffic to the new version first, checking that nothing is broken, then increasing it in steps until you reach the final split. A simple schedule is a 1 to 2 day smoke test with 5 to 10% of traffic on the variant, then 20 to 25%, then 50/50. Once you reach the final split, keep it fixed and read results only from that point on.
Key takeaways
- The first days of a test are for catching bugs, broken tracking and revenue drops, not for finding a winner.
- At each stage, check QA, tracking parity, sample ratio mismatch, guardrail metrics and error rates before you ramp.
- Never use statistical significance to decide when to ramp. Use checks and minimum time.
- Once the split is final, don’t change it. Count results from the moment it became final.
- Ramping a test is about safe measurement. Rolling a winner out to 100% after the test is a separate release step.
Why not launch an A/B test at 50/50 straight away?
Because a new variant is new code, and new code breaks in ways nobody caught in QA. If it breaks at 50/50, half of your paying traffic hits the problem before anyone notices. Typical problems in the first hours of a test:
- Bugs on real devices. A layout that worked in the preview breaks on one browser, one phone size or with one ad blocker.
- Broken tracking. The purchase or lead event doesn’t fire on the variant, fires twice, or sends the wrong value. The test then compares two different rulers.
- A variant that tanks revenue. A new offer, price display or page order can lose money fast. That is a valid test result, but you want to find out on a small share of traffic.
- Payment or form errors. A checkout step or form field that fails on the variant costs real orders and real leads every minute it runs.
- Redirects and flicker. A redirect or a slow script can lose visitors before they ever see the page, which skews who ends up in each group.
Since I started designing for A/B tests at Spiralyze in 2020, I have treated the first days of a test as QA in production, not as results. A ramp limits the damage of anything QA missed.
What does ramping up an A/B test mean?
Ramping means increasing the share of traffic exposed to the test in steps, with checks between each step. There are two common ways to do it:
- Ramp the test, keep the split. Only a slice of visitors enters the test (say 10 or 20%), split evenly between control and variant. Everyone else sees the current page and isn’t counted. Then you widen the slice. Optimizely’s guide to controlled ramping uses this approach: a short ramp stage, then reset results and move to full allocation with the same split.
- Ramp the variant’s share. Everyone enters the test, but the variant starts at 5 to 10% and grows to 50%. This is common on tools that only let you set the split.
Both work as a safety net. The difference matters for analysis: the second way changes the split between control and variant during the test, so data from the early stages must be kept out of the final read (more on that below).
What does a good A/B test ramp schedule look like?
Three stages: a small smoke test, a middle stage, then the final split. The numbers below are my starting point, not a rule. Low-traffic sites need longer stages; very high-traffic sites can move faster.
| Stage | Traffic to the variant | Minimum time | What to check | Move on when |
|---|---|---|---|---|
| 1. Smoke test | 5 to 10% | 1 to 2 days, at least one full day | QA on real devices, every event fires on both versions, error rates, payments and form submits work | No errors, all events present, the planned ratio holds |
| 2. Middle stage | 20 to 25% | 1 to 3 days | Tracking parity, sample ratio mismatch, guardrail metrics, page speed | Guardrails are not collapsing, no sample ratio mismatch |
| 3. Final split | 50% (or the planned split) | The full planned duration, in full weeks | Sample ratio mismatch and guardrails daily, no winner-hunting | The pre-planned sample size is reached |
If you ramp the test slice instead of the variant’s share, the same logic applies: for example 10% of traffic in the test at 50/50, then 100% of traffic at 50/50.
What should you check before each ramp step?
Five things: QA, tracking parity, sample ratio, guardrails and errors. None of them is “is the variant winning?”. Optimizely’s guidance says it directly: statistical significance should not guide ramping decisions.
- QA on live traffic. Open both versions on a real phone and a desktop, through a real ad link, and complete the full path: click, form or cart, payment or submit.
- Tracking parity. Every event that fires on the control fires on the variant, once, with the right value. Compare event counts per visitor between the two versions. A big gap in something like page views per visitor usually means tracking, not behavior.
- Sample ratio mismatch (SRM). Check that the visitor counts match the split you set. Microsoft’s experimentation team describes SRM as a statistically significant difference between the observed ratio and the configured one, and advises not to trust results until the cause is found.
- Guardrail metrics. Revenue per visitor, average order value, refund or cancellation rate, lead quality. You are not looking for a win, only for a collapse.
- Error rates. JavaScript errors, failed payments, failed form submits, 404s and page load time on the variant compared with the control.
Example
Say you set a 50/50 split and after 20,000 visitors the tool shows 10,400 in the control and 9,600 in the variant. That looks close, but a chi-square test on those counts says a gap this size is very unlikely to happen by chance with a true 50/50 split. Something is losing visitors on one side, often a redirect, a script that fails before the tracking loads, or bots filtered unevenly. Find the cause before you ramp further or read any result.
When is it safe to move to the next stage?
When the checks pass, a minimum time has passed, and enough traffic has gone through the variant to trust the checks. My rules of thumb:
- Minimum time: at least one full day per stage, so the variant sees morning, evening and night traffic. Two days is better for the smoke test.
- Minimum traffic: enough visitors that every tracked step (page view, click, form start, purchase or lead) has fired many times on the variant, and at least a few dozen completed conversions on the variant before you leave the smoke test. On a low-traffic page, that may take longer than two days. Extend the stage rather than skip it.
- All five checks pass: QA, tracking parity, no SRM, guardrails stable, error rates in line with the control.
- Someone is watching: don’t ramp on a Friday evening or right before a sale if nobody can roll back.
When should you roll back an A/B test?
Roll back right away if something is broken, and don’t wait for significance to confirm it. Clear reasons to stop or drop the variant to 0%:
- Payments, form submits or a key step fail on the variant.
- A conversion event is missing, doubled or sends the wrong value.
- Sample ratio mismatch you can’t explain.
- Error rates or load time clearly worse than the control.
- A guardrail collapses, for example orders drop to a fraction of the control’s rate.
After a fix, treat the test as new. Data from before the fix measured a broken version.
Why keep the split steady once you reach 50/50?
Because changing the split mid-test mixes periods with different traffic, and the pooled numbers can mislead you. This is Simpson’s paradox: a variant can win in every period and still look worse (or much better) when the periods are added together, because each version got a different share of each period. Conversion.com shows this with a Friday and Saturday example where the variant wins on both days and still loses on the combined numbers.
There is a second effect. Changing the allocation usually only affects new visitors. People already bucketed stay where they are, so returning visitors, who often convert better, can pile up in one version.
Example
Say the variant runs at 10% in a normal week: the control gets 9,000 visitors at 2.0% and the variant 1,000 at 2.2%. Then you ramp to 50/50 just as a sale starts: 5,000 each, the control at 4.0% and the variant at 4.4%. The variant is 10% better in both weeks. Pooled, the control shows 380 conversions from 14,000 visitors (2.7%) and the variant 242 from 6,000 (4.0%), which looks like a 49% lift. Most of that “lift” is the sale week, which the variant got a bigger share of.
Which data should the final analysis include?
Only data from the moment the split became final, unless your tool uses a method built for changing allocation. In practice:
- Reset or filter. Reset results when you reach the final split, or filter the analysis to start on that date. Optimizely recommends resetting results before moving to full allocation, and LaunchDarkly prompts users to reset results after a ramp to avoid time-based bias.
- Restart the sample count. The planned sample size and duration count from that point, not from the first day of the smoke test.
- Or use a method that handles it. Some stats engines weight results by period or by allocation so that changing splits don’t bias the answer. If yours does, read its documentation and know what it assumes.
The same rule applies to any change after launch: a variant fix or a tracking fix also starts a new experiment. See when to call an A/B test for how to read the results from that point.
How is ramping a test different from rolling out a winner?
Ramping a test is about measuring safely. Rolling out a winner is about releasing safely. They happen at different times and answer different questions.
- Ramping the test happens at the start. The goal is to reach the final split without breaking anything, then measure cleanly.
- Rolling out the winner happens after the test is called. The goal is to ship the winning version to 100% of traffic. Optimizely recommends building the winner into the site’s code rather than pushing the test split to 100% and leaving the experiment running.
If the winning change is large, for example a new checkout, it can still make sense to release it gradually and watch the same guardrails. Just don’t treat that rollout as more test data. The test is over, and its result is the number you report. To turn that number into yearly money, see how to calculate the revenue impact of an A/B test.
If you want tests designed, launched and read this way, see A/B testing or book a call.
FAQ
Should every A/B test be ramped up?
Any test that changes code, tracking, a form, a checkout or a price should be. A pure copy change on a well-tested template carries less risk, but a short smoke test still catches tracking problems, so I ramp almost every test.
How long should the smoke test stage last?
At least one full day, and ideally one to two days, so the variant sees a full daily cycle of traffic. On low-traffic pages, keep it running until each tracked step and a few dozen conversions have come through the variant.
Can I use data from the ramp-up period in the final result?
Not if the split changed. Data from periods with different splits can be biased when pooled. Reset results or start the analysis on the day the final split began, unless your tool uses a method designed for changing allocation.
What is sample ratio mismatch?
It is when the number of visitors in each version doesn’t match the split you set, by more than chance explains. It usually points to a technical problem such as redirects, tracking that fails on one version or uneven bot filtering, and the results shouldn’t be trusted until it is fixed.
Sources
I’m George (Djordje) Peric, a CRO designer, CXL certified in Conversion Optimization and Ecommerce Marketing. I design ads, landing pages and funnels and A/B test them.