How to set up an A/B Test
A lot of configuration goes into running A/B tests on your website. With Umbraco Engage everything can be configured from the backoffice.
In this tutorial, you will go through the necessary steps for setting up and configuring an A/B test on your Umbraco website.
Before you start, it is recommended to read the What is A/B Testing article to get familiar with the concept.
Tutorial Content
Make a plan
The first step to setting up an A/B test on your website is to define a solid plan. It is also important to ensure that your website receives enough traffic to make the tests valid.
This tutorial does not cover defining a plan or what makes a project valid for testing. However, find some general recommendations below.
Set a Clear Objective: Start by defining a specific goal for your A/B test. Knowing what you aim to improve or measure allows you to design the test accordingly. For example, do you want more clicks on a button, a higher conversion rate, or something else?
Ensure a valid amount of data: A significant amount of traffic and conversions is important to produce reliable results from A/B tests. Use a tool like the Risk Optimization Automation Rethink (ROAR) model to determine whether your website is eligible.
Develop a Hypothesis: Based on your objective, create a clear hypothesis. This should include what change you expect and why. The stronger your hypothesis, the more valuable your test results will be.
Once you have confirmed your website is eligible for testing and defined a clear hypothesis and objective, you are ready to set up the test.
Before you start
Every test must belong to a project. If none exist yet, see Create a project before starting the test.
Choose test type
Umbraco Engage enables you to run different types of tests depending on your objective and goal for running the test.
The steps for setting up each test are more or less the same. You can set up the tests on content items in the Content section or from the A/B Testing dashboard in the Engage section. In the table below, you can see which tests can be initialed from which section.
Single Page Test
Multi-page Test
Multi-page Test
Document Type Test
Document Type Test
Split URL Test
Allow segmentation on your Document Types
Only content that uses Document Types configured for segmentation can be used for the tests. See the For Developers box below for more details.
Create the test
The following steps will take you through the initial setup of the test. Use the table above to determine from which backoffice section the test should be created.
Set up a test from the Content section
Navigate to the content item where you want to set up an A/B test.
Open the A/B Tests view.
Click on Create new test.
Set up a test from the Engage section
Navigate to the A/B Testing dashboard in the Engage section.
Click on Add new test.
Configure the test
The following steps guide you through configuring the different parts of the test.
Information
Enter the Title of this test.
Under Part of project, select Choose to pick an existing project. Select + to create a new one.
Enter a Description / hypothesis, ideally focused on the hypothesis the test is based on.

Settings
Under Select a conversion goal, select Choose to pick a goal. Select + to create a new one.
Use Set audience percentage to decide how many visitors take part in the test.
Change the Expected conversion change under Advanced settings if relevant.
Select Save. The Test setup section becomes available once the test is saved.

Test setup
Choose the test type in Select test type.
Single page: Available from the Content section only. No more configuration required.
Multi page: Select the pages to use with Choose under Select multiple pages.
Content type: Select the Document Types to use under Select document types.
Split URL: Available from the Engage section only. Select the pages to use with Choose under Select multiple pages.
Give the new variant a name under Add variants. Select Add to add more variants.
You can edit the variants at a later point.
Select Save.

With all the configurations in place, editing the variants added to the test is next.
Edit the test variants
When you have configured and set up the test, it will be in draft mode until you decide to start the actual test.
It is now time to start building the variant(s). This is primarily done by adding custom code to each variant. If you are running a Single Page Test you can edit the content on the page using the split-view functionality.
Add variations to content (Single Page Test only)
Navigate to the content item in the Content section.
Open the A/B Tests view.
Select the test.
Click on Edit next to the variant you want to make changes to.

You can now edit the content item in split view. Only fields that have been configured for segmentation can be edited. If you leave a field blank, it will use the values from the original version.

Select Save and preview to preview the changes before publishing.
Save and publish the content variant.
Add custom code
Navigate to the content item in the Content section.
Open the A/B Tests view.
Select the test.
Select CSS / JS next to the variant you want to make changes to.

Add an optional description to the variant.
Expand CSS or JavaScript under Custom code and add your code.

Select Save.
Select Preview next to the variant to check the changes.
Access the test from the A/B Testing dashboard in the Engage section.
Select CSS / JS next to the variant you want to make changes to.

Add an optional description to the variant.
Expand CSS or JavaScript under Custom code and add your code.

Select Save.
Select Preview next to the variant to check the changes.
Start the test
You have now set up all the variants and the test is ready to be started.
Open the test in the Engage section.
Select Schedule test to pick a start date, or Start test to start right away.
Next steps
The A/B test has been set up on your website and will start gathering data and results. This concludes the tutorial.
Find resources for the next steps in the articles listed below.
Monitor the test
Monitor and keep an eye on your A/B test while it is running. You can track the results throughout the process, but it is encouraged not to make changes until the test is complete.
Finish the test
Learn how to wrap up an A/B test once a result has been found.
Last updated
Was this helpful?