What to do when an A/B test is flat (inconclusive)

What does a flat A/B test result actually mean?

It means the test could not detect a difference at the size it was built to detect. It does not mean the two versions perform the same. Every test is sized for a minimum detectable effect (MDE). Evan Miller’s calculator defines the MDE as the smallest effect that will be detected a set share of the time, which is the test’s power. Effects smaller than that can easily come out flat.

Statisticians have a short phrase for this, used as the title of a well-known BMJ note by Altman and Bland: absence of evidence is not evidence of absence. A flat test is an absence of evidence.

That last sentence is the useful part. A flat result tells you the change did not do what you hoped, at the size you hoped. It doesn’t tell you the idea is worthless.

What should you check before trusting a flat result?

Check that the test was able to find a difference in the first place. A broken test often looks flat, because noise or bad data hides any real effect. Run through these before you decide anything:

• Sample ratio mismatch. The visitor counts should match the split you set. If they don’t, something is dropping or adding visitors on one side, and the result can’t be trusted in either direction. See sample ratio mismatch for how to check it.
• Tracking parity. Every conversion event should fire once on both versions, with the right value. A purchase event missing on one browser can pull a winning variant down to flat.
• Enough sample. The test should have reached the sample size you planned, counted from the last change to the test. A test stopped early is underpowered, not flat. See when to call an A/B test.
• Exposure. Did visitors actually see the change? A new section far below the fold, a change only on one step that few people reach, or a variant that flickered back to control will dilute any effect.
• Novelty and primacy effects. Returning visitors sometimes react to anything new, clicking more at first (novelty) or resisting a change to something familiar (primacy). Plot the daily difference. If it starts large and fades, or starts negative and recovers, the full-period average may hide the settled effect.
• Outside events. A sale, an outage or a campaign change during the test can swamp a small effect for both versions.

If any check fails, you don’t have a flat result. You have an invalid test. Fix the cause and run it again.

Should you look at segments when a test is flat?

Yes, but treat what you find as ideas for the next test, not as winners. Slicing a flat test by device, traffic source, new or returning, or country almost always turns up one slice where the variant “won”. The more slices you check, the more likely one of them looks significant by chance alone.

How I handle segments in a flat test:

• Decide key segments before launch. If mobile versus desktop matters, say so in the hypothesis. A pre-planned segment is far more credible than one found afterwards.
• Look for a reason. A segment result is more believable when there is a clear cause, for example the change only affected a layout that differs on mobile.
• Check the segment’s own sample. A segment is a fraction of the test, so its result is much noisier.
• Test it again. If a segment looks promising, write a new hypothesis for that audience and run a test sized for it.

How do you decide what to do after a flat test?

Pick one of three paths: keep the control, ship the variant for reasons other than lift, or test a bolder version. The table below is the rule of thumb I use.

Two things not to do: keep extending the test until it crosses significance, and quietly report the variant as a winner. Both make your test history less trustworthy.

What does “iterate bolder” mean in practice?

Keep the same problem and hypothesis, but make the change big enough that visitors can’t miss it. Small changes tend to have small effects, and small effects need a lot of traffic to detect. Ways to go bolder:

• Move it up. Put the change on the first screen instead of halfway down the page.
• Make it bigger. Instead of one review line under the button, a full proof block with rating, count and two short quotes.
• Change more of the same idea. If a new headline was flat, rewrite the whole first screen around the ad’s promise: headline, image and first action together.
• Change the offer, not the wording. A different bundle, guarantee or first step often moves more than a different way of describing the same offer.

A bolder test also helps with sample size. A larger expected effect needs far fewer visitors to detect, so the test answers sooner. See A/B test sample size for why.

What can you learn from a flat test?

What the result rules out, what to try next, and whether your test plan was realistic. I write three lines in the test log for every flat result:

1. What it rules out. “Repeating the ad’s headline alone did not lift revenue per visitor by 10% or more on mobile.”
2. What still might be true. “The first screen may still be the problem. The headline alone may not be enough.”
3. What comes next. “Test a full first-screen rewrite: headline, image and first action.”

Over time, the log also shows patterns. If many tests come out flat, the MDE may be too ambitious for your traffic, or the changes may be too small. Both are fixable: plan bolder tests, or put your traffic on fewer, bigger bets. When you pick the next one, ICE prioritization helps, and a clear hypothesis makes the next result easier to learn from.

If you want tests designed to give clear answers, see A/B testing or book a call.

FAQ

Sources

• Evan Miller: Sample Size Calculator
• Altman and Bland, BMJ (1995): Absence of evidence is not evidence of absence
• Evan Miller: How Not To Run an A/B Test
• Microsoft Research: Diagnosing Sample Ratio Mismatch in A/B Testing