How to Split Test In Four Easy Steps. If you do not split test in digital marketing, you are wrong. At Nerds Do It Better we split test every single thing that we can. We split test ads, emails, landing pages, coffee selection at the office (just kidding, kind of), promotions and much more.  We know that it is not helpful to us, or to the client to guess or say “we think so”. Split testing holds us to scientific certainty and we like that.

What is split testing?

Split testing (also called A/B testing) is a type of test that uses two different versions of a variable and runs them against each other at the same time following stringent scientific testing principles. Variables can include email layout, ad copy, landing page forms, unique selling points, and much more. Basically, if it is possible to create two different versions of something, than it is possible to split test. The power of split testing comes from the use of the scientific method which provides the ability to tell with scientific certainty what does and does not work. Imagine being able to confidently say, the next time your boss hovers over your desk and demands you continue to use his “genius” marketing idea – you tested the idea and know for a scientific FACT that it hurt sales. The scientific method gives you that confidence. This method involves a hypothesis, test, learn methodology which I explain in the following section.

How to split test:

Here is how we split test at Nerds Do It Better.


1. Define a conversion

A conversion is anything that you want a visitor to do with your digital message. It is totally up to your business goals but we often prefer for this to be an action, and it must be measurable. Some examples of conversions that our clients have used in the past include visitors filling out forms, making purchases, requesting more information or clicking on an ad. Make the conversion important to your business goals. As an example there is no point in measuring how  long of visitor  stays on the page if it doesn’t help your business goals.

2.  Create a hypothesis

Once you know your conversion, the next step is to create a hypothesis about what you would like to test. If your conversion is, for example. a visitor filling out a form you may want to test two different forms and see which the visitor is more likely to fill out. One form could have more fields to fill out and one form could have less fields to fill out. Your hypothesis may be that visitors are more likely to fill out a shorter form because longer forms are intimidating. Your hypothesis can be whatever you like, as long as it can be tested. In a split test there is always a control and an experiment, it is on you to decide which is which.

Helpful hint: An easy way to decide what is the control and what is the experiment is to test against your current website. For example, if you are testing a new website layout compared to your current website layout, the new layout will be the experiment and the current layout will be the control. Once we have a hypothesis here at NDIB we put it in a word document describing the experiment plan (called the Testing Documentation). This document contains the hypothesis, the control, the experiment, the conversion goal, any necessary images, and the start and estimated end date. We attach this document to Google spreadsheet that we use to track all of our experiments.

Newbie Learning Moment:

I am going to go on a brief tangent to save you time and frustration. A common mistake for newcomers is to split test very small changes. When I first started split testing I made this mistake  myself. I had read how Amazon changed one word on their page and increased conversions by absurd percentages and how other large companies changed a button from “shop now” to “buy now” and increased conversions dramatically. However, when I tried these small changes I couldn’t move the needle. It took me a couple of tries to figure it out, so I will save you the trouble. Unless you have large amounts of traffic (over 500 visitors per day) you will need to make dramatically different versions for split testing. The more dramatic the differences in the control and experiment the faster the split tests run, If you have less than 500 visitors on your page per day and when you start, and you don’t make dramatically different control and experimental versions, you will never reach the 95% confidence level and it will drive you crazier than Charlie Sheen drinking pure tiger blood.


3. Run the experiment until you reach 95% confidence

Experimental conditions can vary by experimental platforms, but generally we use Mailchimp for email split testing, UnBounce for landing page split testing (copy and layout), and adWords and Chad Summerhill’s tool for ad split testing. When I say experiments vary by platforms I mean that with Mailchimp it is just a one off batch and you get the results almost immediately, but with something like adWords the tests may run for a couple of weeks. You need to learn about the settings in each one platform.  No matter what the experiment the conditions must remain the same. Here is what you need to focus on to run the split test:

a. Random assignment of the audience receiving the control and the experiment. Each different experimental platform will allow you to tune your experiment to allow a truly random split test.

b. The experiment is performed in real-time. Sending out one ad now and one in two weeks provides invalid conclusions. This is because the world changes quickly and you need to see how the experiment and control happen in real-time.

c. Always split test to 95% confidence, this eliminates the likelihood that the results happened by chance. Anything lower than 95% and you are taking chances with your conclusions. I have heard stories of wild swings where it seemed like something was going to work and then it didn’t and if only they had waited for the 95%! I have found generally what happens is the split test moves slowly in the beginning and then there is a giant upswing and the experiment concludes quickly.

4.  Learn

Depending on the experiment we may check it daily or weekly. Once we see that the result is 95% we review the results and see if the control or the experiment is the winner. We record the results in the excel document and also the Testing Documentation. Recording the results helps us make better hypothesesis ( what is the plural of this) in the future.  However, it is not time to pop the champagne yet, because you never stop testing. Once you have the winner from the experiment you want to immediately test against it and see if you can beat it. The idea is to Always Be Testing (ABT) so that you are continually improving.

How are you split testing?

Adam Lundquist

@adamlundquist | founder of Nerds Do It Better

Adam Lundquist, is a Harvard-educated agency owner and former radio shock jock from Santa Barbara radio. When Adam published one of his interviews on a newly-formed platform called YouTube, it became one of the first viral videos, sparking his initial interest in the power of the internet, as well as internet marketing. Adam has been featured on traditional media like MTV, VH1, and the Best Damn Sports Show. He’s taken his knowledge of traditional media translated it successfully into new media as a featured author in Search Engine Journal, PPC Hero, and WordStream, amongst others.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>


To Get You More Customers!

We utilize an advanced ROI driven marketing methodology developed at Harvard and put it to work for you to generate more customers on a consistent basis in a cost effective way.