All articles
A/B testing

Why changing the next page can ruin your landing page A/B test

Short answer

A landing page A/B test usually measures leads or sales, and those happen on later pages: the form, the quiz, the checkout. If one of those later pages changes for only one version of the test, the result is invalid, because you no longer know whether the landing page or the later change caused the difference. If the later page changes for both versions in the middle of the test, the test can still be read, but only by splitting the data at the date of the change, and the result may not hold. The fix is simple: freeze every page after the one you test until the test ends, and log any change that has to happen.

Key takeaways

  • A landing page test is measured at the end of the funnel, so every page after it is part of the test.
  • A change on a later page that only one version sees makes the test invalid. Restart it.
  • A change on a later page that both versions see splits the test in two. Read each part on its own.
  • Freeze the funnel while a test runs: no edits to the form, quiz, checkout or thank you page.
  • Keep a change log with dates, so you can spot the problem when a result looks strange.

Why the next page is part of your test

You test the landing page, but you measure what happens later. In a lead generation funnel, the landing page sends people to a form or a quiz, and the conversion is the submitted lead. In ecommerce, the product page sends people to the cart and checkout, and the conversion is the order. The number you compare between version A and version B is created on those later pages.

So an A/B test does not only compare two landing pages. It compares two full paths: landing page A plus everything after it, against landing page B plus everything after it. A clean test only works if the only difference between the two paths is the thing you changed on purpose. Change anything else on one path and the test now has two differences, and the data cannot tell you which one did the work.

Case 1: the next page changes for only one version

This makes the test invalid. It happens more often than people think. Version B links to a new form because the designer built a matching one. A developer adds a field to the form, but only on the URL that version B uses. A quiz step is removed for the new landing page because it "fits the new flow better".

Now version B differs from version A in two places: the landing page and the form. If B wins, you can not say the new landing page won. Maybe the new page did nothing and the shorter form did all the work. Maybe the new page was worse and the form covered it up. In statistics this is called a confound: a second change that moves together with the one you are testing.

Example, made-up numbers

A landing page test runs for two weeks. Version A converts at 4.0%, version B at 5.0%, a 25% lift. Then someone notices that version B sent people to a 3-field form, while version A still used the old 6-field form. A 25% lift is within what a shorter form alone can do. The test says nothing about the landing page. The right call is to fix the form so both versions use the same one, and run the test again.

It also works the other way. If the change made the later page worse for version B, a good landing page can look like a loser. You would throw away a winner without knowing it.

Case 2: the next page changes for both versions, mid-test

This does not break the test by itself, but it splits it in two. Say the checkout gets a new payment option on day 9 of a 21-day product page test. Both versions now send people to the new checkout. The comparison is still fair on any given day, because A and B see the same later pages.

The problem is that you now have two different tests glued together: days 1 to 8 with the old checkout and days 9 to 21 with the new one. Two things can go wrong:

  • The change helps one version more than the other. If version B brings in more hesitant buyers, a better checkout may help B more than A. The lift you see after day 9 is then a mix of the landing page and how it works with the new checkout. This is called an interaction.
  • Your sample plan is off. If the new checkout raises the baseline conversion rate, the effect size and the number of visitors you need change too. A test planned with the sample size calculator before the change may end up too short or too long.

How to read it: split the results at the date of the change and look at both parts. If B wins in both parts by a similar amount, the result is probably solid. If B wins only before or only after the change, treat the test as inconclusive and run it again on the new funnel. Large experimentation teams at Microsoft found that real interactions between tests are rare, but they do happen, and a big change to a shared page is where they are most likely.

Quick guide: is my test still valid?

What changed during the testValid?What to do
A later page, for one version onlyNoMake both versions use the same later pages and restart the test.
A later page, for both versionsPartlySplit the results at the change date. Trust the result only if it holds in both parts.
Ad targeting or traffic source, for bothPartlySame as above. Check that the traffic split is still even with an SRM check.
The tested page itself, mid-testNoAny edit to A or B starts a new test. Restart and count from the edit.
Nothing after the tested pageYesRead the test as planned, once it reaches its sample size.

How to prevent it: freeze the funnel

The simplest rule I use: while a test runs, nothing after the tested page changes. Before a test starts, I write down every page in the path, from the ad to the thank you page, and agree with the team that those pages stay as they are until the readout.

  1. Map the full path. Ad, landing page, form or quiz, checkout, thank you page, and any emails or calls that count as the conversion.
  2. Check that both versions use the same later pages. Same URLs, same form fields, same scripts. Click through both versions yourself before launch.
  3. Announce a freeze. Tell designers, developers and the ads team which pages are locked, and until when.
  4. Keep a change log. If something must change, a bug fix or a legal update, write down what changed, where and when.
  5. Read the test with the log open. If a change happened, split the data at that date as described above.
  6. Test the later page next. If the form or checkout needs work, that is a great next test. Run it after this one ends, or as a separate test on its own traffic.

A freeze costs a few weeks of patience. Skipping it can cost you a full test, or worse, a wrong decision that you roll out to all your traffic. If you already have results that look too good or make no sense, check the change log first, then the traffic split, then the timing of the readout.

When to test the whole path instead

Sometimes the landing page and the form really do belong together, for example a new offer that needs a new quiz. Then test them as one change on purpose: version A is the old landing page plus the old form, version B is the new landing page plus the new form. The result tells you whether the new path wins. It will not tell you which part did the work, and that is fine, as long as you name the test for what it is. I did this for a solar installer: a new landing page and its 4-step quiz were tested together against the old quote page, with the same ad sending traffic to both. The full story is in the solar quiz landing page case.

Want tests you can trust?

I plan, design and run A/B tests from the ad to the readout, and I keep the funnel clean while they run, so every result tells you what actually earned the money. See my A/B testing service →

FAQ

Can I change my checkout during a landing page A/B test?

Only if both versions get the same change, and then you need to split the results at the date of the change. If only one version gets the new checkout, the test is invalid and should be restarted.

What is a confound in an A/B test?

A second difference between the versions that you did not mean to test, like a different form behind version B. It makes it impossible to say which change caused the result.

Should I restart an A/B test after a change?

Restart it if the change touched only one version or the tested page itself. If the change hit both versions on a later page, you can split the data at the change date and only trust a result that holds in both parts.

Can I test the landing page and the form together?

Yes, as one test of the full path. Version B is the new page plus the new form. You learn whether the new path wins, but not which part caused it.

Sources

George Peric, CRO designer

Written by George Peric

I’m George (Djordje) Peric, a CRO designer, CXL certified in Conversion Optimization and Ecommerce Marketing. I design websites, landing pages, funnels and ads, and A/B test them.

Let’s talk

Let’s make your next test a winner.

Tell me about your funnel. I’ll point out quick wins, tell you what I’d test first, and say honestly whether I’m the right fit.