Information on how to deploy Umbraco in a Load Balanced scenario and other details to consider when setting up Umbraco for load balancing.

Overview

Configuring and setting up a load balanced server environment requires planning, design and testing. This document should assist you in setting up your servers, load balanced environment and Umbraco configuration.

This document assumes that you have a fair amount of knowledge about:

• Umbraco

• IIS 10+

• Networking & DNS

• Windows Server

• .NET5+

Design

These instructions make the following assumptions:

• All web servers can communicate with the database where Umbraco data is stored

• You are running Umbraco 9.0.0 or above

• You will designate a single server to be the backoffice server for which your editors will log into for editing content. Umbraco will not work correctly if the backoffice is behind the load balancer.

There are three design alternatives you can use to effectively load balance servers:

2. Each server hosts copies of the load balanced website files and a file replication service is running to ensure that all files on all servers are up to date

3. The load balanced website files are located on a centralized file share (SAN/NAS/Clustered File Server/Network Share)

You will need a load balancer to do your load balancing.

How Umbraco load balancing works

In order to understand how to host your site it is best to understand how Umbraco's flexible load balancing works.

The following diagram shows the data flow/communication between each item in the environment:

The process is as follows:

• Administrators and editors create, update, delete data/content on the backoffice server

• These events are converted into data structures called "instructions" and are stored in the database in a queue

• Each front-end server checks to see if there are any outstanding instructions it hasn't processed yet

• When a front-end server detects that there are pending instructions, it downloads them and processes them and in turn updates it's cache, cache files and indexes on its own file system

• There can be a delay between content updates and a front-end server's refreshing, this is expected and normal behaviour.

Scheduling and server role election

Although there is a backoffice server designated for administration, by default this is not explicitly set as the "Scheduling server". In Umbraco there can only be a single scheduling server which performs the following 3 things:

• Keep alive service - to ensure scheduled publishing occurs

• Scheduled tasks - to initiate any configured scheduled tasks

• Scheduled publishing - to initiate any scheduled publishing for documents

Automatic Server Role Election

Umbraco will automatically elect a "Scheduling server" to perform the above services. This means that all of the servers will need to be able to resolve the URL of either: itself, the Backoffice server, the internal load balancer, or the public address.

There are two server roles:

• SchedulingPublisher - Usually this is the backoffice instance.

• Subscriber - These are the scalable front-end instances - not recommended to be used for backoffice access.

Each instance will be allocated a role by the automatic server role election process, but they can also be set explicitly (recommended)

For example, In the following diagram the node f02.mysite.local is the elected "Scheduling server". In order for scheduling to work it needs to be able to send requests to itself, the Backoffice server, the internal load balancer or the public address. The address used by the "Scheduling server" is called the "umbracoApplicationUrl".

By default, Umbraco will set the "umbracoApplicationUrl" to the address made by the first accepted request when the AppDomain starts. It is assumed that this address will be a DNS address that the server can resolve.

For example, if a public request reached the load balancer on www.mysite.com, the load balancer may send the request on to the servers with the original address: www.mysite.com. By default the "umbracoApplicationUrl" will be www.mysite.com. However, load balancers may route the request internally under a different DNS name such as "f02.mysite.local" which by default would mean the "umbracoApplicationUrl" is "f02.mysite.local". In any case the elected "Scheduling server" must be able to resolve this address.

In many scenarios this is fine, but in case this is not adequate there's a few of options you can use:

Common load balancing setup information

The below section applies to all ASP.NET load balancing configurations.

Server Configuration

This section describes the configuration options depending on your hosting setup:

Data Protection

The replacement for Machine Keys in ASP.NET Core are called Data Protection. You will need to setup data protection to the same keys on all servers, without this you will end up with view state errors, validation errors and encryption/decryption errors since each server will have its own generated key.

Session State and Distributed Cache

Because Umbraco in some cases uses TempData, your setup needs to be configured with a distributed cache.

Logging

There are some logging configurations to take into account no matter what type of load balancing environment you are using.

Testing

Your staging environment should also be load balanced so that you can see any issues relating to load balancing in that environment before going to production.

You'll need to test this solution a lot before going to production. You need to ensure there are no windows security issues, etc... The best way to determine issues is have a lot of people testing this setup and ensuring all errors and warnings in your application/system logs in Windows are fixed.

Ensure to analyze logs from all servers and check for any warnings and errors.

Unattended upgrades

When upgrading it is possible to run the upgrades unattended.

FAQs

Here's some common questions that are asked regarding Load Balancing with Umbraco:

Question> Why do I need to have a single web instance for Umbraco admin?

TL:DR You must not load balance the Umbraco backoffice, you will end up with data integrity or corruption issues.

The reason you need a single server is because there is no way to guarantee transactional safety between servers. This is because we don't currently use database level locking, we only use application (c#) level locks to guarantee transactional data integrity which is only possible to work on one server. If you have multiple admins saving and publishing at once between servers then the order in which this data is read and written to the database absolutely must be consistent otherwise you will end up with data corruption.

Additionally, the order in which cache instructions are written to the cache instructions table is important for LB, this order is guaranteed by having a single admin server.

Question> Can my SchedulingPublisher backoffice admin server also serve front-end requests?

Yes. There are no problems with having your SchedulingPublisher backoffice admin server also serve front-end request.

However, if you wish to have different security policies for your front-end servers and your back office servers, you may choose to not do this.

Ensure you read the Load Balancing overview and general Azure Web Apps documentation before you begin - you will need to ensure that your ASP.NET Core & logging configurations are correct.

Azure Requirements

• 2 x App service plans with 1 x web app in each:

  • One for the backoffice (Administrative) environment

  • One for your scalable public-facing environment (Public)

• 1 x SQL server that is shared with these 2 web apps

The setup above will allow for the proper scaling of the Administrative and Public web apps.

The App Service plan with the Administrative web app should only be scaled up. The reason for this is that the web app needs to stay as a single instance.

The App Service plan with the Public web app can be scaled both out and up.

Lucene/Examine configuration

The single instance Backoffice Administrative Web App should be set to use SyncedTempFileSystemDirectoryFactory.

The multi-instance Scalable Public Web App should be set to use TempFileSystemDirectoryFactory.

Umbraco TEMP files

When an instance of Umbraco starts up it generates some 'temporary' files on disk. In a normal IIS environment, these would be created within the folders of the Web Application. In an Azure Web App, we want these to be created in the local storage of the actual server that Azure happens to be used for the Web App. So we set this configuration setting to 'true' and the temporary files will be located in the environment temporary folder. This is required for both the performance of the website as well as to prevent file locks from occurring due to the nature of Azure Web Apps shared files system.

{
    "Umbraco": {
        "CMS": {
            "Hosting": {
                "LocalTempStorageLocation" : "EnvironmentTemp"
            }
        }
    }
}

Host synchronization

Umbraco runs within a .NET Host.

When a host restarts, the current host 'winds down' while another host is started. This means there can be more than one live host during a restart. Restarts can occur in many scenarios including when an Azure Web App auto-transitions between hosts, you scale the instances or you utilize slot swapping.

Some file system based services in Umbraco such as the Published Cache and Lucene files can only be accessed by a single host at once. Umbraco manages this synchronization by an object called IMainDom.

By default Umbraco v9.4 & 9.5 uses a system-wide semaphore locking mechanism. This mechanism only works on Windows systems and doesn't work with multi-instance Azure Web Apps. We need to swap it out for an alternative file system based locking mechanism by using the following appSetting. With Umbraco v10+ FileSystemMainDomLock is the default setting.

{
    "Umbraco": {
        "CMS": {
            "Global": {
                "MainDomLock" : "FileSystemMainDomLock"
            }
        }
    }
}

Apply this setting to both the SCHEDULINGPUBLISHER Administrative server and the SUBSCRIBER scalable public-facing servers.

Steps to set-up an environment

1. Create an Azure SQL database

2. Install Umbraco on your backoffice administrative environment and ensure to use your Azure SQL Database

3. Install Umbraco on your scalable public-facing environment and ensure to use your Azure SQL Database

4. Test: Perform some content updates on the administrative environment, ensure they work successfully in that environment, then verify that those changes appear on the scalable public-facing environment

5. Fix the backoffice environment to be the SCHEDULINGPUBLISHER scheduling server and the scalable public-facing environment to be SUBSCRIBERs - see Setting Explicit Server Roles

Scaling

Do not scale your backoffice administrative environment this is not supported and can cause issues.

The public-facing subscriber Azure Web Apps can be manually or automatically scaled up or down and is supported by Umbraco's load balancing.

Deployment considerations

Since you have 2 x web apps, when you deploy you will need to deploy to both places - There are various automation techniques you can use to simplify the process. That is outside the scope of this article.

If the file system on your servers isn't performing any file replication then no Umbraco configuration file changes are necessary. However Media will need to be configured to use a shared location such as Blob storage or S3.

Depending on the configuration and performance of the environment's local storage you might need to consider and the .

Synchronised File System

If the file system on your servers is performing file replication then the Umbraco temporary folder (~/umbraco/Data/TEMP) must be excluded from replication.

If the file system on your servers is located on shared storage you will need to configure Umbraco to locate the Umbraco temporary folder outside of the shared storage.

Replication techniques

A common way to replicate files on Windows Server is to use [DFS](https://msdn.microsoft.com/en-us/library/windows/desktop/bb540031(v=vs.85), which is included with Windows Server.

Additional DFS resources:

There are other alternatives for file replication out there, some free and some licensed. You'll need to decide which solution is best for your environment.

Non-replicated files

When deploying Umbraco in a load balanced scenario using file replication, it is important to ensure that not all files are replicated - otherwise you will experience file locking issues. Here are the folders and files that should not be replicated:

• ~/umbraco/Data/TEMP/*

• ~/umbraco/Logs/*

  • This is optional and depends on how you want your logs configured (see below)

If for some reason your file replication solution doesn't allow you to not replicate specific files folders (which it should!!) then you can use an alternative approach by using virtual directories.

The following is not the recommended setup but it is a viable alternative:

• Copy the ~/umbraco/Data/TEMP directory to each server, outside of any replication areas or to a unique folder for each server.

• Create a virtual directory (not a virtual application) in the ~/umbraco/Data/ folder, and name it TEMP. Point the virtual directory to the folder you created in step 2.

• You may delete the ~/umbraco/Data/TEMP folder from the file system - not IIS as this may delete the virtual directory - if you wish.

IIS Setup

IIS configuration is pretty straightforward with file replication. IIS is only reading files from its own file system like a normal IIS website.

Mixture of standalone & synchronised

In some scenarios you have a mixture of standalone and synchronised file systems. An example of this is Azure Web Apps where the file system isn't replicated between backoffice and front end servers but is replicated between all front end servers, in this configuration you should follow the steps for synchronised file systems.

Examine Directory Factory Options

• The TempFileSystemDirectoryFactory allows Examine to store indexes directly in the environment temporary storage directory, and should be used instead of SyncTempEnvDirectoryFactory mentioned above.

• The SyncedTempFileSystemDirectoryFactory enables Examine to sync indexes between the remote file system and the local environment temporary storage directory, the indexes will be accessed from the temporary storage directory. This setting is needed because Lucene has issues when working from a remote file share so the files need to be read/accessed locally. Any time the index is updated, this setting will ensure that both the locally created indexes and the normal indexes are written to. This will ensure that when the app is restarted or the local environment temp files are cleared out that the index files can be restored from the centrally stored index files.

Advanced techniques

Umbraco v8+ uses Serilog for logging. When load balancing Umbraco consideration should be given as to how the log files from each server will be accessed.

There are many Serilog Sinks available. One of these may be appropriate to store logs for all servers in a central repository such as Azure Application Insights or Elmah.io.

For more information, see .

This describes some more advanced techniques that you could achieve with flexible load balancing

The election process that runs during the startup of an Umbraco instance determines the server role that instance will undertake.

There are two server roles to be aware of for flexible load balancing:

• SchedulingPublisher - The Umbraco instance usually used for backoffice access, responsible for running scheduled tasks.

• Subscriber - A scalable instance that subscribes to content updates from the SchedulingPublisher server, not recommended to be used for backoffice access.

Explicit SchedulingPublisher server

It is recommended to configure an explicit SchedulingPublisher server since this reduces the amount of complexity that the election process performs.

The first thing to do is create a couple of small classes that implement IServerRoleAccessor one for each of the different server roles:

public class SchedulingPublisherServerRoleAccessor : IServerRoleAccessor
{
    public ServerRole CurrentServerRole => ServerRole.SchedulingPublisher;
}

public class SubscriberServerRoleAccessor : IServerRoleAccessor
{
    public ServerRole CurrentServerRole => ServerRole.Subscriber;
}

then you'll need to replace the default IServerRoleAccessor for the your custom registrars. You'll can do this by using the SetServerRegistrar() extension method on IUmbracoBuilder from a Composer.

// This should be executed on your single `SchedulingPublisher` server
builder.SetServerRegistrar<SchedulingPublisherServerRoleAccessor>();

// This should be executed on your `Subscriber` servers
builder.SetServerRegistrar<SubscriberServerRoleAccessor>();

Now that your subscriber servers are using your custom SubscriberServerRoleAccessor class, they will always be deemed 'Subscriber' servers and will not attempt to run the automatic server role election process or task scheduling.

By setting your SchedulingPublisher server to use your custom SchedulingPublisherServerRoleAccessor class, it will always be deemed the 'SchedulingPublisher' and will always be the one that executes all task scheduling.

Subscriber servers - Read-only database access

In some cases infrastructure admins will not want their front-end servers to have write access to the database. By default front-end servers will require write full access to the following tables:

• umbracoServer

• umbracoNode

This is because by default each server will inform the database that they are active and more importantly it is used for task scheduling. Only a single server can execute task scheduling and these tables are used for servers to use a server role election process without the need for any configuration. So in the case that a subscriber server becomes the SchedulingPublisher task scheduler, it will require write access to all of the Umbraco tables.

In order to have read-only database access configured for your front-end servers, you need to implement the Explicit SchedulingPublisher server configuration mentioned above.

Now that your subscriber servers are using your custom SubscriberServerRoleAccessor class, they will always be deemed 'Subscriber' servers and will not attempt to run the automatic server role election process or task scheduling. Because you are no longer using the default ElectedServerRoleAccessor they will not try to ping the umbracoServer table.

Controlling how often the load balancing instructions from the database are processed and pruned

The configurations can be adjusted to control how often the load balancing instructions from the database are processed and pruned.

Below is shown how to do this from a JSON configuration source.

{
    "Umbraco": {
        "CMS": {
            "Global": {
                "DatabaseServerMessenger": {
                    "MaxProcessingInstructionCount": 1000,
                    "TimeBetweenPruneOperations": "00:01:00",
                    "TimeBetweenSyncOperations": "00:00:05",
                    "TimeToRetainInstructions": "2.00:00:00"
                }
            }
        }
    }
}

Options:

• TimeToRetainInstructions - The timespan to keep instructions in the database; records older than this number will be pruned.

• MaxProcessingInstructionCount - The maximum number of instructions that can be processed at startup; otherwise the server cold-boots (rebuilds its caches)

• TimeBetweenSyncOperations - The timespan to wait between each sync operations

• TimeBetweenPruneOperations - The timespan to wait between each prune operation

These setting would normally be applied to all environments as they are added to the global app settings. If you need these settings to be environment specific, we recommend using environment specific appSetting files.