How to calculate the revenue impact of an A/B test
George Peric · Conversion designer · 3 min readShort answer
Multiply the difference in revenue per visitor between the variant and the control by the number of visitors who will see the change in a year. That gives you the expected extra revenue per year. Use a conservative lift and remove anything temporary, so the number survives scrutiny.
Key takeaways
- Use revenue per visitor (value per session), not conversion rate, as the base of the calculation.
- Yearly impact = (variant revenue per visitor − control revenue per visitor) × yearly visitors who will see it.
- Report a conservative number: the lower end of the confidence interval, not the best case.
- Watch for novelty effects, seasonality and changes that only affect part of the traffic.
What is the formula for A/B test revenue impact?
Yearly revenue impact = (revenue per visitor in the variant − revenue per visitor in the control) × visitors per year that will see the change.
Δ RPV = RPVvariant − RPVcontrol
Yearly impact = Δ RPV × yearly visitors exposed
Revenue per visitor (RPV) is total revenue divided by visitors. In lead generation and call funnels it is often called value per session: revenue divided by sessions. It is the best single number for a test because it already combines how many people convert and how much each conversion is worth.
Why use revenue per visitor instead of conversion rate?
Because conversion rate can go up while revenue goes down. A variant can push more people to convert by attracting lower-value orders or lower-quality leads. Revenue per visitor catches that, because the value of each conversion is already inside it.
In call and lead funnels, also add a quality guardrail, such as the share of calls that qualify or leads that sales can reach. A win on volume that loses on quality is not a win.
A worked example
Example
Say a landing page gets 100,000 sessions a month. The control earns $2.00 per session and the variant earns $2.10. These are round, made-up numbers to show the math, not a client result.
| Step | Value |
|---|---|
| Control value per session | $2.00 |
| Variant value per session | $2.10 |
| Difference per session | $0.10 (+5%) |
| Sessions per month | 100,000 |
| Extra revenue per month | $0.10 × 100,000 = $10,000 |
| Extra revenue per year | $10,000 × 12 = $120,000 |
Now make it conservative. If the confidence interval for the lift runs from +2% to +8%, use +2%: $2.00 × 2% = $0.04 per session, × 100,000 × 12 = $48,000 a year. Report both: about $120,000 expected, $48,000 conservative.
How do you make the number conservative?
Use the low end of the confidence interval for the lift instead of the observed lift. The observed lift is the single most likely value, but the true effect could be lower. If your testing tool reports a 95% confidence interval of +3% to +14%, calculating with +3% gives a number that will hold up when finance looks back in six months.
I usually report two numbers: the expected impact (observed lift) and the conservative impact (low end). Decisions get made faster when nobody has to argue about whether the number is inflated.
Five mistakes that inflate A/B test revenue
- Using conversion rate lift on total revenue. A 10% conversion lift on a page that only affects a third of revenue is not a 10% revenue lift.
- Counting all traffic when only some sees the change. If the change is only on mobile, or only on one landing page, only that traffic counts.
- Ignoring the novelty effect. Some changes win early because they are new. Look at whether the lift holds in the second half of the test.
- Annualizing a seasonal peak. A test that ran during your busiest month will overstate a normal month. Use average traffic, not peak traffic.
- Stopping as soon as it looks good. Peeking and stopping early makes false winners. Decide the duration up front and run at least two full weeks.
How should you report the result?
One sentence that a CFO can repeat: “The variant raised revenue per visitor by X%, which is worth about $Y per year at current traffic (conservatively $Z).” Then add the guardrails that held and what you will test next. Percentages start the conversation, money ends it.
FAQ
What is value per session?
Value per session is total revenue divided by the number of sessions. It is the same idea as revenue per visitor and is common in lead generation and call funnels, where revenue comes from leads or calls rather than orders.
Should I calculate A/B test impact with conversion rate or revenue per visitor?
Revenue per visitor. It includes both how many people convert and how much each conversion is worth, so it does not reward changes that bring in more but cheaper conversions.
How do I annualize an A/B test result?
Multiply the difference in revenue per visitor by the monthly visitors who will see the change, then by 12. Use average monthly traffic rather than a peak month, and a conservative lift.
Is the observed lift the real lift?
Not exactly. It is the most likely value, but the true effect sits somewhere in the confidence interval. Use the low end of the interval for a conservative estimate.
Sources
Conversion designer, CXL certified in Conversion Optimization and Ecommerce Marketing. I design ads, landing pages and funnels and A/B test them.