A/B testing, also known as split testing, is a way of working out which of two campaign options is the most effective in terms of encouraging opens or clicks.
​

In an A/B test you set up two variations of the one campaign and send them to a small percentage of your total recipients. Half of the test group is sent Version A, while the other half gets Version B. The result, measured by the most opens or clicks, determines the winning version. This is then sent to the remaining subscribers.

When an A/B test is in progress you can't change the campaign, but you can cancel it before it is sent to your entire list.

You can turn an email campaign into an A/B test at any point before sending it. To get started:

When creating A/B tests it’s important to think about what you are trying to improve or optimise. Is your intention to get more people to read your email, drive more traffic to your website, or encourage recipients to click on a specific call to action? Your primary goal determines the winning metric for your A/B test.

Alternatively, you could:

Choose your goal and click Continue.

Next, choose what you want to test in Version A and Version B.

We will recommend one of these options based on the primary goal you have selected, but you can choose any configuration you wish.

Once you've made your choice, click Save.

For this test, campaign versions A and B are identical except for the subject line (and preview text if you wish). A subject line test is recommended to increase open rates, however, varying the subject is also a valid tactic when the goal is increased clicks.

For example, you could test to see if a generic subject gets more open than a longer subject line that's more specific or interesting.

Alternatively, you could:

Sender details are important because many people will not open an email if they don't recognise who it's from. With the From name test, you can use a different name and email address for Version A versus Version B, or just change one or the other.

The best approach depends on your relationship with the subscriber. Consider if they are more likely to recognise an individual's name, your company name, or the product name your campaigns are about.

This is to test different elements of the campaign itself, for example, section titles, article length, calls-to-action, header images and more. You might even test two completely different designs to see which one gets the most clicks.

Campaigns for this type of test can be created using one of your saved templates or designed externally and imported (if that option is available to you). If you're doing a design test, follow the on-screen instructions to set up Version A first. You can’t begin to work on the design for Version B until the content for Version A is in place.

If you're using the same template and only making minor changes to content, save time by copying the content from Version A to create Version B. Click Design email for Version B and then select Copy version A.

When both versions of the campaign are set up you'll be prompted to choose the subscriber list, or lists, to send to. If new subscribers are added to a list after the A/B test has started they will not be sent the campaign. Any subscribers added while a test is in progress will have to be targeted separately.

It's important to test and check for errors before you start an A/B test because you can't make changes when it is in progress.

To double-check everything you've set up so far, we summarise it for you in a campaign checklist. If you need to change something just click the Edit buttons on the right of the snapshot.

If you're testing email content using a template-based campaign, the checklist is where you can check the plain text version of your emails. What looks great as HTML might need a little work in plain text. Below the email design preview, in the campaign checklist, click Manage plain text version to see how your text version looks and make any necessary changes.

When you're satisfied that everything is good to go click Test and define delivery. Run a Quick test to send yourself previews of both versions or click Full test to run a fully automated design and spam test.

Once all elements of your A/B test are in place, you’re ready to send. Click Prepare to send in the top right corner.

The next step is selecting the size of your test group, deciding how the winner will be chosen (only for tests where the goal is clicks on a specific link), and setting a length of time to run the test.

Use the slider to define a test group, which should be a small subset of your recipients, say 20–30%. Recipients in the test group are selected randomly. Half of them will be sent Version A while the other half are sent Version B. The remaining recipients will be sent the winning version.

When you start a test we send all emails (Versions A and B) to the test group at the same time. The remainder is held back for when the winner is decided.

There’s a brief reminder of the way you’ve configured the A/B test. Click Change if you’ve made an error and want to redo the setup.

Once you’re done configuring the testing group split, click Continue to decide when to send the A/B test.

The length of time you can run an A/B test is from one hour, up to five days. After this time passes, the system will look at the data collected against Versions A and B and decide the winner (with reference to the goal of the test). The winning version is then sent to all remaining recipients. Or you can manually select a winner while the test is in progress.

You can start the A/B test immediately or select Schedule for later to pick a time and date.

If you schedule the test to start later, we'll send you an email notification when it starts.

You can cancel an A/B test any time before a winner is selected. When you cancel an in-progress A/B test:

There are multiple places from which you can cancel an in-progress A/B test. The first is from the Overview or Campaigns section of your account. In the Sending section, to the right of the A/B test campaign you want to cancel, click Cancel.

Finally, you can cancel an A/B test from its results page. To do so:

When you start the cancellation process, you will see a link to download a CSV file. This file contains the subscribers who have and haven't been sent a version of your campaign. If you made an error in your campaign and want to send an updated version, this data can help you to figure out who to send the updated version to.

To do so:

While a test is in progress you can monitor results with the A/B test report, which looks like this:

The graph makes it easy to see which version of the campaign is winning, according to the performance metric you've chosen: open rate, total unique clicks, or total clicks on a selected link. Below the graph you can also see which version is winning for the two metrics you didn't select.
​

For some A/B tests there's a clear winner before the test time is up. On the report page, pictured above, you can see the time remaining and a link to manually select the winner. Click this to choose the winner yourself and end the test early.

After a winner has been decided, through test completion or manual selection, the remaining recipients will be sent the winning version and a full campaign report will become available. The A/B test results are included in the full report so you can revisit them when designing your next campaign.
​

If you let the test run to completion we'll send you an email notification as soon as it's finished. If the two versions are tied at the end of the test, version A will be sent to the rest of the list.