How to set up an A/B Test
A lot of configuration goes into running A/B tests on your website. With Umbraco Engage everything can be configured from the backoffice.
In this tutorial, you will go through the necessary steps for setting up and configuring an A/B test on your Umbraco website.
Before you start, it is recommended to read the What is A/B Testing article to get familiar with the concept.
Tutorial Content
Make a plan
The first step to setting up an A/B test on your website is to define a solid plan. It is also important to ensure that your website receives enough traffic to make the tests valid.
This tutorial does not cover defining a plan or what makes a project valid for testing. However, find some general recommendations below.
Set a Clear Objective: Start by defining a specific goal for your A/B test. Knowing what you aim to improve or measure allows you to design the test accordingly. For example, do you want more clicks on a button, a higher conversion rate, or something else?
Ensure a valid amount of data: A significant amount of traffic and conversions is important to produce reliable results from A/B tests. Use a tool like the Risk Optimization Automation Rethink (ROAR) model to determine whether your website is eligible.
Develop a Hypothesis: Based on your objective, create a clear hypothesis. This should include what change you expect and why. The stronger your hypothesis, the more valuable your test results will be.
Once you have confirmed your website is eligible for testing and defined a clear hypothesis and objective, you are ready to set up the test.
Choose test type
Umbraco Engage enables you to run different types of tests depending on your objective and goal for running the test.
The steps for setting up each test are more or less the same. You can set up the tests on content items in the Content section or from the A/B Testing dashboard in the Engage section. In the table below, you can see which tests can be initialed from which section.
Single Page Test
Multi-page Test
Multi-page Test
Document Type Test
Document Type Test
Split URL Test
Allow segmentation on your Document Types
Only content that uses Document Types configured for segmentation can be used for the tests. See the For Developers box below for more details.
Create the test
The following steps will take you through the initial setup of the test. Use the table above to determine from which backoffice section the test should be created.
Set up a test from the Content section
Navigate to the content item where you want to set up an A/B test.
Open the A/B Tests view.
Click on Start a test.
Set up a test from the Engage section
Navigate to the A/B Testing dashboard in the Engage section.
Click on Start new test.
Configure the test
The following steps guide you through configuring the different parts of the test.
Step 1: Information
Give the test a Title.
Select which Project the test is a part of or create a new project.
Give the test a Description ideally focused on the hypothesis the test is based on.
Step 2: Setup Test
Select which Test Type should be used.
Single Page: No more configuration required.
Multiple Pages: Select the pages to be used in the test.
Document Type: Select the Document Type for the test.
Split URL: Select the pages to be used for the test.
Decide if visitors are assigned to the test when visiting the specific test page or any page on the website.
Give the new variant a name.
You can edit the variants at a later point.
Step 3: Settings
Select or create a conversion goal for the test.
Set the Audience percentage.
Change the Expected conversion rate under Advanced settings if relevant.
Save the test.
Select Next step to get an overview of the test.
With all the configurations in place, editing the variants added to the test is next.
Edit the test variants
When you have configured and set up the test, it will be in draft mode until you decide to start the actual test.
It is now time to start building the variant(s). This is primarily done by adding custom code to each variant. If you are running a Single Page Test you can edit the content on the page using the split-view functionality.
Add variations to content (Single Page Test only)
Navigate to the content item in the Content section.
Open the A/B Tests view.
Select the test.
Click on Edit next to the variant you want to make changes to.
You can now edit the content item in split view. Only fields that have been configured for segmentation can be edited. If you leave a field blank, it will use the values from the original version.
Preview the changes before publishing.
Save and publish the content variant.
Add custom code
Navigate to the content item in the Content section.
Open the A/B Tests view.
Select the test.
Click on Edit next to the variant you want to make changes to.
Click on the Edit variant button below the Page Title.
Add an optional description to the variant.
Add custom CSS and JavaScript.
Preview the changes.
Save the changes.
Start the test
You have now set up all the variants and the test is ready to be started.
Open the test in the Engage section.
Schedule a date for the test to start or Start the test right away.
Next steps
The A/B test has been set up on your website and will start gathering data and results. This concludes the tutorial.
Find resources for the next steps in the articles listed below.
Monitor and keep an eye on your A/B test while it is running. You can track the results throughout the process, but it is encouraged not to make changes until the test is complete.
Learn how to wrap up an A/B test once a result has been found.
Last updated
