A/B testing is one of the main features of Umbraco Engage. A/B testing allows you to test your website and specific pages with two sets of configurations. Umbraco Engage supports both the novice and the experienced A/B tester.

This documentation section will help you out with A/B Testing, and covers the following topics:

• A brief overview of what A/B testing is.

• The kind of A/B Tests that Umbraco Engage supports.

• How you can set up the A/B test.

• How the results during an A/B test can be analyzed.

• How a visitor is assigned the A, B, or n-variant.

Three different types of A/B tests are available:

• Single-page tests to test a specific page within Umbraco.

• A way to test multiple pages at the same time.

• Test an entire Document Type and all of the pages using this type.

• Split URL testing. This enables you to select two or multiple pages and serve one of these URLs per visitor.

The A/B tests are set up within an Umbraco context. You can reuse properties that are already created. Umbraco Engage is also aware of the structure of your website and concepts like Document Types.

When setting up the A/B test, you can select the type of test you want to set up.

A Single-Page test can only be started in the A/B test Content App and not from the Engage section.

The Single-Page test allows an editor to start an A/B test without a single line of code.

When you select the Single-Page test type you can create two or more variants.

The first variant is always the original content and the published page. Variant B is the first variant that can be created and with the button 'Add a variant' more alternatives can be added. More variants mean that a test should run for longer to become reliable.

Split view editing

The variant can be given a name and if you click on Edit you will open up the split view editor:

On the left side, the original content is shown (Original) and the variant is shown on the right. In this side-by-side editing mode, you can edit the content for your variant.

Setting up the Document Type for split-view editing

Some properties are inactive (visually indicated because they are greyed out). Which properties you can edit is specified in the setup of the properties of your Document Type.

Specifying when segmentation is allowed can be done per property using the Allow segmentation value.

In the overview of the Document Type, you will see if properties can be segmented as they will have the Vary by segments label:

Add CSS or JavaScript

Sometimes you cannot adjust a specific property because it was not configured when Umbraco was set up. In those cases, you can use the CSS/JavaScript field to add a code-snippet to make these adjustments. The best way is to do it via property editing in the split view edit mode. You do not have to write any CSS or JavaScript code.

To do this click CSS/JavaScript in the A/B Testing editor bar:

This will give the editor a popup where CSS and JavaScript can be entered:

These lines of code will automatically be inserted at the bottom of the page.

Use the Save & preview button to make sure all works as expected. This can also be done via the "Save & preview" option in the editor bar.

Once the variant is set up, finish the A/B Test via the Back to A/B test button in the editor bar.

Finish the steps in the Settings up the A/B test article to verify and start your A/B Test.

Content rendering

Because of the unique implementation of Umbraco Engage, the content will automatically be updated for this variant.

The algorithm will determine which variant of the propety needs to be rendered.

It goes too far to give a total overview of what A/B testing is. There are some great resource out there that can inspire you and can go in real depth what A/B testing is.

For now we will stick with this explaination: A/B testing is a methodical way of testing (and hopefully improving) your website. By serving multiple versions of a webpage, content block, or website to different visitors, gathering data on which version performs better for a specific goal.

Why should you A/B test your website?

Your website is always underperforming its potential. Your website will never have a conversion rate of 100% and probably not even near that conversion rate. All visitors do exactly what you want them to do. By testing specific parts of your website you can experiment and hopefully increase your conversion rate. The internet gives you the ability to test with different variants of your website and see what happens.

A/B Testing in Umbraco

There are already a lot of great A/B testing tooling out there (Google Experiments, Optimizely, Visual Website Optimizer). They do a really great job on A/B testing but they lack a key feature that Umbraco Engage offers: full integration within Umbraco. This allows you to integrate A/B testing in your daily workflow instead of an add-on on top of your website. This means:

• You can A/B test your content in the same user-friendly way as Umbraco does, allowing you and your editors to start A/B testing.

• The A/B test is rendered automatically without any additional line of code. This means that you don't have to reserve time to "implement A/B testing". It is there and you can start testing immediately. Also that means there is no content flickering like in many other tools.

• You can use all the resources that Umbraco gives you. You can reuse images of the media library, you can pick contentblock as you are used, and you do not have to worry about changing URL's while doing a test.

• Within Umbraco Engage you will see that A/B tests are running, so your fellow editors are aware of this and will not ruin your experiments

• Because we use first-party analytics and cookies you will track data across all your visitors and not only on a subset of customers which do not have any ad- and cookie block tooling

• Our A/B test is context aware and will make sure that the A/B test is render everywhere correctly. For example; if you are testing a catchy product title it will automatically not render only that new title on the page itself, but also on your overview, your homepage, your shopping cart or at any other place you're using the title. This is impossible with any other tool!

Some great resources

Many books explain the importance of A/B testing and why you should start testing your website today. A short list of the books that inspired the Umbraco Engage team:

• Dan Siroker and Pete Koomen. A/B Testing. The most powerful way to turn clicks into customers. Hoboken, New Jersey: John Wiley & Sons, Inc., 2013. Print

• Chris Goward. You should test that! Indianapolis, Indiana: John Wiley & Sons, Inc., 2013. Print

• Will Kurt. Bayesian statistics the fun way. San Fransisco, William Pollock, 2019. Print

Split URL testing in Umbraco Engage allows you to compare different versions of a page to see which one performs better.

To set up a Split URL test, you must create multiple versions of a page. Each version can have different designs, text, or buttons based on the options setup in Umbraco.

You can start a Split URL test from the Engage section or within the Content App. The type is selected in step 2 of the setup.

When you set up the A/B test, you can send 50% of visitors to page A and 50% to page B. This way, you can see which page helps achieve the goal of the test better.

Split URL tests are particularly useful for optimizing elements like checkout workflows, where small changes can impact conversions.

Setting up an A/B test in Umbraco Engage requires no code or need for any external tooling.

Starting A/B tests can be done in two ways:

• Initiate a Single Page, Multiple Page, or Document Type test via the Content App on the specific page,

• Initiate Multipage and Document Type tests via the Engage section.

Set up the test

Open the Umbraco content tree and select the A/B tests Content App on the page.

Make sure that you have access to the Content App. These permissions can be specified per Document Types or User Groups.

When you open the Content App you will have an overview of all the A/B tests that are running or finished. You also have the option to start a new test.

Configure the A/B test

When you start a new test you will have to specify the following:

• The name of the test.

• Which testing project it should be part of.

• Write a description/hypothesis.

After that, you have to specify the type of test and set up the specific test. It is also possible to preview your variant from here.

As a final step, you need to specify which goal you want to measure, as it is all about optimizing for this goal. Set the audience that you want to include in the test.

Launching the A/B test

Once you have set everything up the test is ready to go. You can preview the final variants and check if everything set up correct.

If that is the case you can start the test immediately, or schedule it for later.

The Multiple Pages test allows you to across multiple pages at once. In Umbraco, you can select the pages you want to test. For all these pages you can specify which type of CSS or JavaScript will be added to the specific variant. The multiple pages test requires you to write (or copy in) some CSS and JavaScript code.

The test type Multiple pages can be started in the Engage section and in the Content App. The type is selected in step 2 of the setup.

The test allows you to select one or more pages within Umbraco. On all these pages the A/B Test will render the additional CSS and JavaScript you enter. The CSS and JavaScript must not create any side effects on these pages. This is a manual job that cannot be automated with Umbraco Engage.

Once you have selected the pages you want to test, you can specify one or more variants. For each variant, it is possible to click the Edit button. This will bring up a popup that allows you to write JavaScript or CSS:

In this example, some JavaScript is added to change the page's background color.

You can save and preview whether your code works by clicking Save & preview.

After you have created all variants start your A/B testing as described in the Setting up the A/B test article.

Test all pages using a specific Document Type with this test. Select the Document Type(s) you want to test and Umbraco Engage makes sure the correct CSS and JavaScript is inserted to the correct pages.

The test type Document Type can be started in the Engage section and in the Content App. The type is selected in step 2 of the setup.

The test allows you to select one or more Document Types within Umbraco. On all pages using the selected Document Type(s) the A/B Test will render the additional CSS and JavaScript you will enter. The CSS and JavaScript must not create any side effects on these pages. This is a manual task.

Once you have selected the pages you want to test, you can specify one or more variants. For each variant, it is possible to click the Edit button. This will bring up a popup that allows you to write JavaScript or CSS:

In this example, some JavaScript changes the page's background color.

You can save and preview whether your code works by clicking Save & preview.

After creating all variants, start your A/B testing as written in the Setting up the A/B test article.

There are different ways to preview your A/B Test variants. In this article, we will discuss all options.

During the setup of your new A/B Test

When setting up a new A/B Test there are three options to preview your A/B Test variant. First of all, you open the preview of your variant by clicking on the preview button in the overview of variants:

When editing a specific variant on a single page you can also preview the page. In the top bar, there is a Save & Preview button available:

You can also click on Edit variant which will open a pop-up where you can add some CSS and JavaScript. Also from here, you can open up the preview by clicking on Save & preview:

There are plenty of options to open up the preview.

With all variants set up, you can get an overview before the A/B Test. In that screen, you also have the option to preview all variants:

During a running test

You can preview all the A/B test variants while a test is running. Go to the specific test, and you will have an overview of the current results. You can also preview each variant by clicking the "preview"-button.

The A/B Test will automatically be rendered in your front end. If you have set up a Umbraco Engage will automatically render the correct properties.

If you have set up one of the other tests and added some CSS and JavaScript this will automatically be added to your webpage.

You do not have to do anything to render the A/B test. If you want to run a more advanced A/B test that requires more coding, you can always do that.

You can decide to finish an A/B test at any time. Normally, you would end a test if it has been running long enough and enough data to gain enough confidence is collected.

Umbraco Engage lets you know when the collected data is statistically significant. The same applies to when not enough data is collected.

Even if not enough data is collected you can always decide whether a running A/B Test is complete. In the overview you can see all tests and their performance:

To end a test, click End test. This will open a confirmation dialog asking if you want to pick a winner directly.

You might want to discuss the results with a colleague or client. In that case, it makes sense to stop the A/B test, but complete it by picking a winner at a later point.

If you select Stop and complete now the A/B Test is stopped and you can select which variant was the winner:

You can always pick one of the variants or My test did not result in a winner. After you confirm your selected winner the overview will show the picked winner:

The overview of the A/B tests will also not show that the A/B Test is stopped and a winner was selected:

The default algorithm is based on randomness. When visitors visit the website or a specific page with an A/B test, they get randomly assigned a variant. That variant is stored alongside the visitor's cookie and the same A/B test variant is shown as long as that cookie exists.

Because the algorithm is based on randomness the visitors are not equally distributed between the variants and slight differences can occur.

In the test shown below, the variant distribution is not a perfect 33,333% as you could expect with three variants.

When the A/B test is running it is advised to check the progress regularly. This can be done on the specific page or via the Engage Section.

If you go to the overview you can see all running tests. The overview includes the following information:

• For how long a test has been running?

• How many visitors are included in the test?

• For how long should the test run to produce relevant results?

You can see all test variants when you access the details on a test. You can preview the variants, or disable them. If for example, the conversion rate has decreased in a variant, it could be smart to disable that variant.

The algorithm also signals the leading variant based on the Minimal Detectable Effect (MDE) which uses the 95% confidence level.