Short answer
A good A/B test hypothesis names the evidence, the change, the audience, the expected effect on one metric, and how you will judge it. A simple template: Because [evidence], changing [element] to [new version] for [audience] will [raise or lower] [primary metric]. It worked if [primary metric] moves by at least [minimum effect] and [guardrail] holds, after [sample size per version, in full weeks]. Write it before you design anything.
Key takeaways
- A hypothesis without evidence is a guess. Start from a problem you saw in data, recordings, surveys or reviews.
- Name one primary metric, one or two guardrails and the sample size before launch, inside the hypothesis itself.
- Change one idea per test when you want to learn why something worked. Bundle changes only when you want a bigger effect and accept learning less.
- A strong hypothesis is useful even when the test loses, because it tells you what to try next.
What is an A/B test hypothesis?
An A/B test hypothesis is a short, testable statement of what you will change, why, for whom, and what result would prove it right or wrong. It turns an idea like “let’s try a new headline” into something you can measure and learn from. Optimizely’s guide to hypotheses puts it simply: the difference between a well-formed hypothesis and a guess is data.
A hypothesis also protects the test from you. Once the metric, the minimum effect and the sample size are written down, it is much harder to move the goalposts after launch or to pick a “winning” metric after the fact.
What is a good A/B test hypothesis template?
Use one sentence for the idea and one sentence for how you will judge it. This is the template I use when I design tests:
Copy this template
Because [evidence: what you saw in data, recordings, surveys or reviews],
changing [element] to [new version]
for [audience: traffic source, device, new or returning]
will [raise or lower] [primary metric].
It worked if [primary metric] moves by at least [minimum detectable effect] and [guardrail metric] does not get worse, after [sample size per version], run in full weeks.
Each part has a job:
- Evidence says why you think the problem exists. If you can’t fill it in, you are not ready to test.
- Change says exactly what the variant does differently, specific enough for a designer and a developer to build.
- Audience says who sees it. A change for mobile paid social visitors is a different test from one for all traffic.
- Effect on metric gives a direction and one primary metric. Revenue per visitor is usually the best one for money pages.
- Success rule names the minimum effect, the guardrail and the sample, so the result is judged on rules set before launch. See A/B test sample size for how to get the sample number.
If you prefer the classic “If, then, because” form that Optimizely recommends, it carries the same parts in a different order. The format matters less than having every part filled in.
What makes a hypothesis strong or weak?
A strong hypothesis is specific, based on evidence and measurable. A weak one is vague, based on taste, or impossible to prove wrong.
| Part | Weak | Strong |
|---|---|---|
| Evidence | “I think the page feels cluttered.” | “Recordings show most mobile visitors scroll past the form without starting it.” |
| Change | “Improve the hero section.” | “Replace the brand slogan with the ad’s promise, word for word.” |
| Audience | “Users.” | “Mobile visitors from paid social ads.” |
| Metric | “Engagement will go up.” | “Revenue per visitor will go up.” |
| Success rule | “Let’s see how it goes.” | “At least a 10% relative lift, refund rate flat, after the planned sample in two full weeks.” |
| If it loses | Nothing learned | The problem stays on the list, and the next test tries a different fix for it |
The last row is the real test of a hypothesis. If a loss would teach you nothing, the hypothesis was too vague.
What do good A/B test hypotheses look like? Six examples
Here are six example hypotheses written with the template, for landing pages, forms, checkout and pricing. They are examples to show the format, not results from real tests. Swap in your own evidence and numbers.
Example
1. Landing page headline (message match). Because the top-spend ad promises “free delivery on your first box” and the landing page opens with a brand slogan, changing the headline to repeat the ad’s promise for mobile visitors from that ad will raise revenue per visitor. It worked if revenue per visitor rises by at least 8% and refund rate holds, after the planned sample in two full weeks.
2. Landing page proof. Because the on-page survey shows “Does this actually work?” as the top doubt, and reviews sit at the bottom of the page, moving the star rating and review count next to the main button for all paid traffic will raise click-through to the offer. It worked if revenue per visitor rises by at least 5% and average order value holds.
3. Lead form fields. Because form analytics show the biggest drop at the phone number field, making phone optional for desktop and mobile visitors will raise completed leads. It worked if completed leads rise by at least 10% and the qualified lead rate does not fall.
4. Multi-step form. Because recordings show mobile visitors pausing at a nine-field form and leaving, splitting it into three short steps that start with the easiest question will raise completed submissions on mobile. It worked if mobile submissions rise by at least 10% and lead quality holds.
5. Checkout costs. Because the biggest checkout drop is at the step where shipping cost first appears, showing the shipping cost on the cart page for all visitors will raise completed orders. It worked if revenue per visitor rises by at least 5% and average order value does not fall.
6. Pricing page. Because sales calls keep hearing “I didn’t know which plan to pick”, marking the middle plan as “Most popular” and listing who it is for will raise paid sign-ups from the pricing page. It worked if revenue per visitor from the pricing page rises by at least 8% and refund or downgrade rate holds.
Notice that each one has a single idea, a named audience and a money metric with a guardrail. For more ideas to plug into the template, see A/B test ideas for landing pages.
Where does the evidence for a hypothesis come from?
From what visitors actually do and say, not from opinions in a meeting. Optimizely’s guidance says to avoid hypotheses based on intuition and to begin with a problem, not a solution. The sources I use most:
- Analytics. Funnel drop-off by step, device and traffic source. The biggest drop is usually the best place to look.
- Session recordings and heatmaps. Where people hesitate, rage-click, scroll past or leave.
- Form analytics. Which field people abandon or correct most often.
- On-page surveys and exit polls. One question, such as “What is stopping you today?”, gives you the visitor’s words.
- Reviews, support tickets and sales calls. The doubts and phrases real customers repeat.
- The ad itself. Comparing the ad with the first screen often shows a message mismatch in seconds.
- Past tests. Losers and flat tests are evidence too. See what to do when an A/B test is flat.
- A heuristic review. An expert pass over clarity, proof, friction and speed. It is weaker evidence on its own, so I pair it with data. My CRO audit checklist covers it.
Strong hypotheses usually have two sources pointing at the same problem, for example a funnel drop and survey answers that explain it.
Should a hypothesis test one change or several?
One idea per test if you want to learn why it worked. Several changes together if you want a bigger effect and can accept learning less. Both are valid; just choose on purpose.
- One change (a headline, a field, a cost display) gives a clean answer. If it wins, you know what did it and can reuse the idea on other pages. The cost is that small changes often have small effects, which need a lot of traffic to detect.
- Bundled changes (a new first screen with headline, image, proof and button together) are more likely to move the numbers enough to detect on modest traffic. The cost is that a win doesn’t tell you which part carried it, and a flat result can hide one good change cancelled out by one bad one.
A useful middle ground is to bundle changes that serve the same idea. “Make the first screen continue the ad” can include headline, image and button, because they all test one hypothesis about message match. Changing the headline, the price and the footer at once tests three unrelated ideas, and a result says little about any of them.
Low-traffic sites usually need bolder, bundled tests. High-traffic sites can afford single changes. Once you have several hypotheses ready, rank them with ICE prioritization.
What are common hypothesis mistakes?
Most weak hypotheses fail in one of five ways.
- Solution first. “Test a video on the page” with no problem behind it. Start with the problem, then pick the fix.
- No primary metric, or five of them. With many metrics, one will move by chance. Pick one and name guardrails.
- A metric too far from money. Clicks and scroll depth can rise while revenue falls.
- No success rule. Without a minimum effect and sample size, the test ends whenever someone likes the numbers. See when to call an A/B test.
- Too small to detect. A one-word change on a low-traffic page can run for months without a clear answer.
Over six years and more than 300 A/B tests designed, from Spiralyze to G-Plans to freelance work, the tests that taught the most were the ones with a clear hypothesis, whether they won or not. If you want help turning research into test plans, see A/B testing or book a call.
FAQ
What is a good hypothesis for an A/B test?
One that names the evidence, the exact change, the audience, the expected effect on one primary metric, and the success rule: minimum effect, guardrail and sample size. For example: because the biggest form drop is at the phone field, making phone optional will raise completed leads, as long as lead quality holds.
What is the If, then, because format?
It is a common way to write a test hypothesis: if you make this change, then this metric will move, because of this evidence. It holds the same parts as the template in this post, in a different order.
Should an A/B test hypothesis include a number?
Yes. Name the minimum effect you care about and the sample size you need to detect it. Without a number, there is no clear line between a win and noise, and it becomes tempting to stop the test whenever the result looks good.
Can I test several changes in one hypothesis?
Yes, if they serve one idea, such as making the first screen continue the ad. A win then tells you the idea works but not which part carried it. Unrelated changes in one test make the result hard to learn from.
Sources
I’m George (Djordje) Peric, a CRO designer, CXL certified in Conversion Optimization and Ecommerce Marketing. I design ads, landing pages and funnels and A/B test them.