How to prioritize A/B tests with an ICE score sheet
George Peric · CRO designer · 8 min readShort answer
ICE ranks A/B test ideas by three scores from 1 to 10: Impact (how much it could move the main metric), Confidence (how much evidence says it will work) and Ease (how cheap it is to build and launch). Multiply the three, sort the sheet from highest to lowest, and test from the top. The score only works if every number is tied to clear anchors and real evidence, so agree on those anchors as a team and re-score after every result.
Key takeaways
- ICE = Impact × Confidence × Ease, each scored 1 to 10. I multiply rather than average, so an idea that is weak on any one factor sinks.
- Impact depends on the page’s traffic, how close the page is to the money and how big the change is.
- Confidence must point to evidence: analytics, recordings, surveys or past tests. No evidence means a low score.
- Keep the sheet alive: re-score when new research or a test result comes in, and review it at least once a month.
What is an ICE score?
ICE is a quick way to rank test ideas by three scores: Impact, Confidence and Ease, each from 1 to 10. It is usually credited to Sean Ellis, who used it to rank growth experiments. The point is not precision. The point is to stop arguing about which idea “feels” best and to make everyone show the reasons behind their pick.
- Impact: if this works, how much will it move the main metric?
- Confidence: how sure are you that it will work, based on evidence?
- Ease: how cheap and fast is it to design, build, approve and launch?
How do you calculate the ICE score?
Multiply the three scores: ICE = Impact × Confidence × Ease, which gives a number from 1 to 1,000. Some teams average them instead, which keeps the result on the same 1 to 10 scale.
I prefer multiplying. With an average, an idea that is trivial but very easy (Impact 2, Ease 10) can climb above a real idea that is harder to build. Multiplying punishes any factor that is close to zero, which is what I want: a test that can’t move the metric, or that nobody believes in, should not reach the top just because it takes ten minutes to build. The example sheet below shows the difference.
How do you score Impact?
Impact is a mix of three things: how much traffic sees the change, how close the page is to the money, and how big the change is. A bold change on a low-traffic page and a tiny change on the checkout can both end up in the middle.
| Score | Impact | Confidence | Ease |
|---|---|---|---|
| 8 to 10 | A page most paying visitors see, one step from purchase or lead, and a change visitors will clearly notice (first screen, offer, form) | Several sources agree: analytics, recordings, survey answers, and a similar past test that won | A copy or layout change one person can design and build in a day or two, no approvals needed |
| 4 to 7 | A mid-traffic page, or a high-traffic page with a moderate change (proof block, button copy near the decision) | One solid source, such as a clear drop in analytics or a pattern in recordings | A few days of design and development, one approval round |
| 1 to 3 | A low-traffic page, far from the conversion, or a change most visitors won’t notice (footer, small wording tweak) | An opinion, a competitor doing it, or “best practice” with nothing from your own data | New features, back-end work, legal or brand review, or several teams involved |
In practice, an Impact of 8 means “the first screen of the top-spend landing page, a change everyone sees”. An Impact of 3 means “a wording change in a section that a fraction of visitors scroll to”.
How do you score Confidence?
Confidence should be a count of evidence, not a feeling. I ask the person scoring it to name the source in the sheet. Good sources, roughly from weaker to stronger:
- Heuristics and best practice: useful to start, but on their own worth 2 or 3.
- Analytics: a big drop at a step, or a large gap between mobile and desktop.
- Session recordings and heatmaps: people hunting for the price, rage-clicking, or missing the button.
- Surveys and customer interviews: visitors naming the doubt in their own words.
- Past tests: a similar change that won on another page or in a related audience.
When two or three of these point the same way, Confidence can go to 8 or above. A CRO audit is the fastest way to collect this evidence before you score anything.
How do you score Ease?
Ease is the full cost of getting the test live: design, development, QA and approvals. Teams often score only the development time and forget the rest. A two-hour build that needs a legal review and a brand sign-off is not a 9. I also count tracking work: if the test needs a new event or a new goal set up, that lowers Ease.
What does an ICE sheet look like?
Here is an example sheet with six hypothetical test ideas for an ecommerce site, scored and sorted by ICE. The ideas and numbers are made up to show the method, not results from a real client. I added the average next to the multiplied score to show how the two differ.
Example
Say a store sells mostly through paid social traffic to one landing page, and an audit found that the headline doesn’t repeat the ad’s promise, the rating sits far below the button, and the mobile button scrolls out of view.
| Test idea | I | C | E | ICE (I×C×E) | Average |
|---|---|---|---|---|---|
| Headline repeats the ad’s promise on the top landing page | 8 | 8 | 9 | 576 | 8.3 |
| Rating and review count next to the button | 7 | 7 | 8 | 392 | 7.3 |
| Sticky call to action on mobile | 6 | 7 | 8 | 336 | 7.0 |
| Split checkout into short steps | 8 | 5 | 3 | 120 | 5.3 |
| New pricing page layout | 7 | 3 | 4 | 84 | 4.7 |
| Change footer link color | 2 | 3 | 10 | 60 | 5.0 |
The headline test goes first. Note the footer idea: with an average it would rank above the pricing page idea, only because it is easy. With multiplication it drops to the bottom, where it belongs.
What columns should a real ICE sheet have?
At minimum: idea, hypothesis, page, metric, I, C, E, score, status and owner. The scores alone are not enough. Without the hypothesis and the evidence, nobody can check the numbers a month later.
| Column | What goes in it |
|---|---|
| Idea | A short name for the change, such as “Rating next to button” |
| Hypothesis | “Because research showed X, changing Y will improve Z for these visitors” |
| Page | The page or step where the test runs |
| Metric | The primary metric, ideally revenue per visitor, plus any guardrail |
| I | Impact, 1 to 10, using the agreed anchors |
| C | Confidence, 1 to 10, with the evidence named in a note |
| E | Ease, 1 to 10, including design, build, QA and approvals |
| Score | I × C × E, calculated by a formula, never typed by hand |
| Status | Backlog, in design, in build, live, won, lost, inconclusive |
| Owner | The one person who moves it to the next status |
I also like a short evidence column next to Confidence and a link to the design file. Before an idea moves to “in build”, check that the page has enough traffic for it with how many visitors you need for an A/B test.
What are the most common ICE scoring mistakes?
The biggest one is score inflation: everyone gives their own idea an Impact of 9. When every idea scores high, the sheet stops sorting anything. The other common mistakes:
- Confidence with no evidence behind it. A 7 that nobody can explain is a guess.
- Ease that counts only development. Approvals and tracking set-up are where tests stall.
- Ignoring traffic. A great idea on a page that needs six months to reach a result is not a high-impact test right now.
- Scoring once and never again. The sheet goes stale and the team quietly stops trusting it.
- Letting the loudest person score. Then ICE just writes down the opinion of whoever runs the meeting.
How do you calibrate ICE scores as a team?
Agree on written anchors, score alone first, then compare and discuss only the big gaps. This is the routine I use:
- Write the anchors down. Put a table like the one above at the top of the sheet, adapted to your site and traffic.
- Pick two reference ideas. Agree that one idea is “an Impact 8” and another is “an Impact 3”, and score new ideas against them.
- Score alone, then compare. Each person scores without seeing the others. Where scores differ by 3 or more, talk it through.
- Make Confidence cite a source. No source, no score above 4.
- Check against results. After a few tests, look at whether high scorers actually won more often, and adjust the anchors.
How does ICE compare to PIE and PXL?
All three rank test ideas, but they differ in how much they rely on judgment. ICE and PIE are quick and subjective. PXL, published by CXL, replaces most of the judgment with yes or no questions about the change and the evidence behind it.
| Framework | Factors | How it scores | Best for | Weak spot |
|---|---|---|---|---|
| ICE | Impact, Confidence, Ease | 1 to 10 each, multiplied (or averaged) | Fast backlogs and small teams | Subjective unless anchors are agreed |
| PIE | Potential, Importance, Ease | 1 to 10 each, averaged | Deciding which pages to test first | No factor for evidence |
| PXL | Noticeability, page traffic, evidence from research, ease | Mostly yes or no questions, added up | Mature programs with many ideas and many opinions | Slower to set up and fill in |
I use ICE with strict anchors for most backlogs, because the team will actually keep it updated. If a team keeps fighting about scores, borrowing PXL’s yes or no questions for Impact and Confidence usually settles it.
How often should you re-score the backlog?
Re-score whenever new evidence arrives, and review the whole sheet at least once a month. New evidence means a test result, a new round of recordings or surveys, a tracking fix or a change in traffic. A loss on one page should lower Confidence for similar ideas. A win should raise it. A shift in ad spend can change which page has the most Impact.
In six years of designing for A/B tests, at Spiralyze, at G-Plans and freelance, the sheets that worked were the ones someone owned and re-scored after every result. For more ideas to put in the sheet, see A/B test ideas for landing pages. If you want a backlog built and run for you, see A/B testing or book a call.
FAQ
Should you multiply or average ICE scores?
Either works if the whole team uses the same method. I multiply, because it pushes down any idea that is weak on one factor, such as an easy change that can’t move the metric. Averaging is easier to read but lets very easy ideas rank too high.
What is a good ICE score?
There is no fixed threshold. The score is only useful for ranking ideas against each other in the same sheet, scored with the same anchors. Test from the top and keep an eye on traffic limits.
Is ICE better than PIE?
ICE has a Confidence factor that rewards evidence, which PIE lacks. PIE is handy for deciding which pages to work on first. Many teams use PIE to pick pages and ICE to rank ideas within them.
Who should score the ideas?
Everyone involved in testing, each scoring alone first: the CRO lead, the designer, a developer for Ease and someone close to the data. Then compare and discuss the big gaps.
Sources
I’m George (Djordje) Peric, a CRO designer, CXL certified in Conversion Optimization and Ecommerce Marketing. I design ads, landing pages and funnels and A/B test them.