The idea of an A/B test is that half of the visitors see version A, the other half version B, and at the end we look at which one performed better. This way the decision rests on how visitors behave, and taste moves into the background.
The test is needed because our instincts are often wrong. A headline that everyone on the team considers the best can easily lose with visitors to a simpler, less clever version.
In ads we most often test the creative or the text. On a website the tested element can be a headline, a button label, the wording of an offer or the length of a form.
What makes a test reliable
- We change only one thing at a time, otherwise we cannot tell what caused the difference
- The two versions run in the same period, for the same audience
- The test runs until enough data has been collected, and we do not stop it after the first few days
- We decide in advance what decides the winner, for example purchases or quote requests
It is worth documenting the result of every test, what we tested and how it turned out. Over time this builds into your own knowledge base of what works with your audience, and the losing versions teach a lot as well.
With little traffic a test is slow to give a result. If a page gets a few dozen visitors a day, the effect of a small change may not show reliably even over weeks. In that case it is worth testing bigger differences, for example two completely different offers, because their effect shows sooner. On smaller sites we therefore prefer to test in the ads, where data builds up faster.
Ad platforms offer built-in testing, and for websites separate tools split the traffic between the versions. The result of a test always applies to your own audience, so a solution that worked for another company may work differently for you.
Testing brings the most when it is continuous. A winning version becomes the starting point of the next test, so the ads and the page improve step by step. That is why we update the testing plan every month, based on the latest results.
