Sample ratio mismatch (SRM): what it is and how to check it
George Peric · CRO designer · 7 min readShort answer
Sample ratio mismatch (SRM) is when the number of visitors in each version of an A/B test differs from the split you configured by more than chance can explain. Check it with a chi-square goodness-of-fit test on the visitor counts. A very small p-value, commonly below 0.01 or stricter, means something in the test is broken: redirects, bots, tracking, crashes or caching. Don’t trust the results until you find the cause, fix it and restart the test.
Key takeaways
- SRM compares observed visitor counts with the configured split. It is a test of the setup, not of the variant.
- Use a chi-square goodness-of-fit test. Thresholds of p < 0.01 or stricter are common; Microsoft’s team uses p < 0.0005.
- A gap that looks small in percent can still be a clear SRM on a large sample.
- Common causes are redirects, bots, late-firing tracking, variant crashes, caching and changing the split mid-test.
- With SRM, don’t read the results at all. Find the cause, fix it, and restart the count.
What is sample ratio mismatch?
Sample ratio mismatch is a difference between the share of visitors each version actually got and the share you told the testing tool to give it, larger than random assignment would produce. If you set a 50/50 split, you expect close to half the visitors in each version. Random assignment is never perfect, so small gaps are normal. SRM is when the gap is too large to be chance.
Microsoft’s experimentation team describes it as the observed ratio being statistically different from the configured one, and says a test with SRM can’t be trusted until the root cause is known. Their article also shows how an SRM can flip a result: in one case the fix turned an apparent result around.
SRM matters because it means the two groups are no longer comparable. Whatever is removing or adding visitors on one side is usually not random. It removes a certain kind of visitor, such as slow-device users, people who bounce fast or bots, and that alone can create a fake win or hide a real one.
How do you check for sample ratio mismatch?
Run a chi-square goodness-of-fit test on the visitor (or user) counts per version, against the split you configured. You only need the counts, not conversions. The steps:
- Take the observed counts for each version, using the unit you randomize on (users or visitors, not page views).
- Work out the expected counts: total visitors × each version’s configured share.
- Compute the chi-square statistic by adding up, for each version, the squared gap between observed and expected, divided by expected.
- Get the p-value from the chi-square distribution with (number of versions − 1) degrees of freedom. A spreadsheet’s CHISQ.TEST function or a free SRM calculator does this for you.
- Compare it with your threshold. Below it, treat the test as having SRM.
χ² = Σ (observed − expected)² ÷ expected
Degrees of freedom = number of versions − 1
What does an SRM check look like with real numbers?
Here is a worked example with a 50/50 split. The numbers are made up for illustration; the math is real.
Example
Say a test is set to 50/50 and has 50,000 visitors: 25,400 in the control and 24,600 in the variant. That is a 50.8% / 49.2% split, which looks close.
Expected count per version: 50,000 × 0.5 = 25,000.
Control: (25,400 − 25,000)² ÷ 25,000 = 160,000 ÷ 25,000 = 6.4
Variant: (24,600 − 25,000)² ÷ 25,000 = 160,000 ÷ 25,000 = 6.4
χ² = 6.4 + 6.4 = 12.8, with 1 degree of freedom.
The p-value is about 0.00035. A gap this size would happen by chance less than once in 2,500 tests with a true 50/50 split. That is below 0.01 and below Microsoft’s stricter 0.0005, so this test has SRM.
Now say the counts were 25,150 and 24,850 instead. Then χ² = 1.8 and the p-value is about 0.18. That gap is well within normal chance, so there is no sign of SRM.
The lesson: percentages can fool you. 50.8% versus 49.2% sounds harmless, but on 50,000 visitors it is a clear warning. On 2,000 visitors, the same percentages would be normal noise. The test accounts for sample size; your eye doesn’t.
If you want to check this in code, Python’s scipy.stats.chisquare([25400, 24600]) returns the same statistic and p-value, and it is the function Lukas Vermeer’s SRM checker documents for its own check.
What p-value threshold should you use for SRM?
Use a stricter threshold than you use for the test result, commonly p < 0.01 or lower. You run the SRM check often, sometimes daily, and on every test. A loose threshold like 0.05 would raise false alarms on roughly one healthy test in twenty each time you look.
- p < 0.01: the threshold Lukas Vermeer’s SRM Checker FAQ says the tool used to flag SRM.
- p < 0.0005: the more conservative cutoff used in Microsoft’s SRM article.
Whichever you pick, pick it before launch and use it on every test. If a test sits just above your threshold but the gap keeps growing day after day, investigate anyway.
What causes sample ratio mismatch?
Almost always a technical problem in how visitors are assigned, served, tracked or filtered. Lukas Vermeer’s taxonomy of SRM causes groups them into assignment, execution, log processing, analysis and interference. These are the ones I check first on websites:
| Cause | What happens | How to spot it |
|---|---|---|
| Redirects | In a split URL test, the variant redirects to a new page. Some visitors leave or fail during the redirect before tracking fires, so the variant loses visitors. | Variant count is low; the gap is larger on slow connections and mobile |
| Bots and filtering | Bots are bucketed but filtered unevenly, or a variant changes behavior so a bot filter flags more real users on one side. | Gap changes when you include or exclude filtered traffic |
| Late-firing tracking | The variant loads slower or fires the exposure event later, so fast leavers are never counted on that side. | Gap is larger for bounced visitors and slow devices |
| Variant crashes or errors | A script error on one browser stops the variant, or the tracking, from running. | Gap is concentrated in one browser, device or app version |
| Browser or CDN caching | A cached page is served without the test script, or with the wrong version, for some visitors. | Gap appears after a deploy or differs between new and returning visitors |
| Ramping plus returning users | The split changed mid-test. Visitors already bucketed stay in their version, so cumulative counts don’t match the final split. | Gap appears right after a ramp step and is driven by returning visitors |
| Wrong analysis start or filters | The analysis starts before one version was live, or uses a filter that applies to one version only. | Counts match when you remove the filter or change the start date |
The ramp case needs a note. If you ramped the variant from 10% to 50%, the cumulative counts are supposed to differ from 50/50, and comparing them against 50/50 will always flag SRM. Check SRM only on data from the moment the final split began. My post on how to ramp up an A/B test covers why the final read should start there too.
What should you do when a test has SRM?
Stop reading the results, find the cause, fix it, and restart the test. In order:
- Don’t trust any metric from the test. Not the primary metric, not the guardrails, not segments. The groups are no longer comparable.
- Find where the gap lives. Microsoft’s article suggests checking whether the mismatch is localized to certain segments and which direction it goes. Break the counts down by browser, device, traffic source, new or returning, and day.
- Check the usual suspects in the table above: redirects, bots, tracking timing, errors, caching and ramp changes.
- If the cause is unclear and the test is live, turn it off. Optimizely’s guidance on SRM says to investigate the cause and, if you can’t find it, stop the experiment to limit exposure.
- Fix and restart. After the fix, count results from zero. Data from before the fix measured a broken setup. See when to call an A/B test on counting from the last change.
Prevention is cheaper. Check SRM daily from the first day, not only at the end, and run an A/A test when you set up a new tool or a new type of test. Optimizely recommends both: early checks and an A/A test to catch configuration problems.
Is there a free SRM calculator?
Yes. Lukas Vermeer’s SRM Checker has a free manual calculator. You enter the observed count and expected ratio for each version, and it shows a p-value and whether there is a possible mismatch. It handles more than two versions and uneven splits. Many testing platforms also run SRM checks automatically; the same site keeps a list of platforms that do.
A spreadsheet works too: CHISQ.TEST with the observed counts and the expected counts gives the p-value directly.
SRM checks are one of the first things I look at on any test, before any result. If you want tests set up and checked this way, see A/B testing or book a call.
FAQ
What is sample ratio mismatch in A/B testing?
It is when the number of visitors in each version of a test differs from the configured split by more than chance can explain. It points to a technical problem, such as redirects, bots, tracking or caching, and the results should not be trusted until it is fixed.
How do I calculate sample ratio mismatch?
Use a chi-square goodness-of-fit test on the visitor counts per version against the expected counts from your split. A free SRM calculator, CHISQ.TEST in a spreadsheet or scipy.stats.chisquare in Python gives you the p-value.
What p-value indicates SRM?
Thresholds of 0.01 or stricter are common. Lukas Vermeer’s SRM Checker used 0.01, and Microsoft’s experimentation team uses 0.0005. Use a stricter level than for the test result, because you check SRM often.
Can I still use the results of a test with SRM?
No. The groups are no longer comparable, so any result, including segments, can be biased in either direction. Find and fix the cause, then restart the test and count from the fix.
Sources
I’m George (Djordje) Peric, a CRO designer, CXL certified in Conversion Optimization and Ecommerce Marketing. I design ads, landing pages and funnels and A/B test them.