Umbraco v8+ uses Serilog for logging. When load balancing Umbraco consideration should be given as to how the log files from each server will be accessed.
There are many Serilog Sinks available. One of these may be appropriate to store logs for all servers in a central repository such as Azure Application Insights or Elmah.io.
For more information, see SeriLog Provided Sinks.
This describes some more advanced techniques that you could achieve with flexible load balancing
The election process that runs during the startup of an Umbraco instance determines the server role that instance will undertake.
There are two server roles to be aware of for flexible load balancing:
SchedulingPublisher
- The Umbraco instance usually used for backoffice access, responsible for running scheduled tasks.
Subscriber
- A scalable instance that subscribes to content updates from the SchedulingPublisher server, not recommended to be used for backoffice access.
These new terms replace 'Master and Replica', in Umbraco versions 7 and 8.
It is recommended to configure an explicit SchedulingPublisher server since this reduces the amount of complexity that the election process performs.
The first thing to do is create a couple of small classes that implement IServerRoleAccessor
one for each of the different server roles:
then you'll need to replace the default IServerRoleAccessor
for the your custom registrars. You'll can do this by using the SetServerRegistrar()
extension method on IUmbracoBuilder
from a Composer.
Now that your subscriber servers are using your custom SubscriberServerRoleAccessor
class, they will always be deemed 'Subscriber' servers and will not attempt to run the automatic server role election process or task scheduling.
By setting your SchedulingPublisher server to use your custom SchedulingPublisherServerRoleAccessor
class, it will always be deemed the 'SchedulingPublisher' and will always be the one that executes all task scheduling.
This description pertains only to Umbraco database tables
In some cases infrastructure admins will not want their front-end servers to have write access to the database. By default front-end servers will require write full access to the following tables:
umbracoServer
umbracoNode
This is because by default each server will inform the database that they are active and more importantly it is used for task scheduling. Only a single server can execute task scheduling and these tables are used for servers to use a server role election process without the need for any configuration. So in the case that a subscriber server becomes the SchedulingPublisher task scheduler, it will require write access to all of the Umbraco tables.
In order to have read-only database access configured for your front-end servers, you need to implement the Explicit SchedulingPublisher server configuration mentioned above.
Now that your subscriber servers are using your custom SubscriberServerRoleAccessor
class, they will always be deemed 'Subscriber' servers and will not attempt to run the automatic server role election process or task scheduling. Because you are no longer using the default ElectedServerRoleAccessor
they will not try to ping the umbracoServer table.
If using SqlMainDomLock on Azure WebApps then write-permissions are required for the following tables for all server roles including 'Subscriber'.
umbracoLock
umbracoKeyValue
SQL Server Replica databases cannot be used as they are read-only without replacing the default MainDomLock with a custom provider.
The configurations can be adjusted to control how often the load balancing instructions from the database are processed and pruned.
Below is shown how to do this from a JSON configuration source.
Options:
TimeToRetainInstructions
- The timespan to keep instructions in the database; records older than this number will be pruned.
MaxProcessingInstructionCount
- The maximum number of instructions that can be processed at startup; otherwise the server cold-boots (rebuilds its caches)
TimeBetweenSyncOperations
- The timespan to wait between each sync operations
TimeBetweenPruneOperations
- The timespan to wait between each prune operation
These setting would normally be applied to all environments as they are added to the global app settings. If you need these settings to be environment specific, we recommend using environment specific appSetting
files.
Information on how to deploy Umbraco in a Load Balanced scenario and other details to consider when setting up Umbraco for load balancing
Information on how to deploy Umbraco in a Load Balanced scenario and other details to consider when setting up Umbraco for load balancing.
Configuring and setting up a load balanced server environment requires planning, design and testing. This document should assist you in setting up your servers, load balanced environment and Umbraco configuration.
This document assumes that you have a fair amount of knowledge about:
Umbraco
IIS 10+
Networking & DNS
Windows Server
.NET5+
It is highly recommended that you setup your staging environment to also be load balanced so that you can run all of your testing on a similar environment to your live environment.
These instructions make the following assumptions:
All web servers can communicate with the database where Umbraco data is stored
You are running Umbraco 9.0.0 or above
You will designate a single server to be the backoffice server for which your editors will log into for editing content. Umbraco will not work correctly if the backoffice is behind the load balancer.
There are three design alternatives you can use to effectively load balance servers:
Each server hosts copies of the load balanced website files and a file replication service is running to ensure that all files on all servers are up to date
The load balanced website files are located on a centralized file share (SAN/NAS/Clustered File Server/Network Share)
You will need a load balancer to do your load balancing.
In order to understand how to host your site it is best to understand how Umbraco's flexible load balancing works.
The following diagram shows the data flow/communication between each item in the environment:
The process is as follows:
Administrators and editors create, update, delete data/content on the backoffice server
These events are converted into data structures called "instructions" and are stored in the database in a queue
Each front-end server checks to see if there are any outstanding instructions it hasn't processed yet
When a front-end server detects that there are pending instructions, it downloads them and processes them and in turn updates it's cache, cache files and indexes on its own file system
There can be a delay between content updates and a front-end server's refreshing, this is expected and normal behaviour.
Although there is a backoffice server designated for administration, by default this is not explicitly set as the "Scheduling server". In Umbraco there can only be a single scheduling server which performs the following 3 things:
Keep alive service - to ensure scheduled publishing occurs
Scheduled tasks - to initiate any configured scheduled tasks
Scheduled publishing - to initiate any scheduled publishing for documents
Umbraco will automatically elect a "Scheduling server" to perform the above services. This means that all of the servers will need to be able to resolve the URL of either: itself, the Backoffice server, the internal load balancer, or the public address.
There are two server roles:
SchedulingPublisher
- Usually this is the backoffice instance.
Subscriber
- These are the scalable front-end instances - not recommended to be used for backoffice access.
These new terms replace 'Master and Replica', in Umbraco versions 7 and 8.
Each instance will be allocated a role by the automatic server role election process, but they can also be set explicitly (recommended)
For example, In the following diagram the node f02.mysite.local is the elected "Scheduling server". In order for scheduling to work it needs to be able to send requests to itself, the Backoffice server, the internal load balancer or the public address. The address used by the "Scheduling server" is called the "umbracoApplicationUrl".
By default, Umbraco will set the "umbracoApplicationUrl" to the address made by the first accepted request when the AppDomain starts. It is assumed that this address will be a DNS address that the server can resolve.
For example, if a public request reached the load balancer on www.mysite.com
, the load balancer may send the request on to the servers with the original address: www.mysite.com
. By default the "umbracoApplicationUrl" will be www.mysite.com
. However, load balancers may route the request internally under a different DNS name such as "f02.mysite.local" which by default would mean the "umbracoApplicationUrl" is "f02.mysite.local". In any case the elected "Scheduling server" must be able to resolve this address.
In many scenarios this is fine, but in case this is not adequate there's a few of options you can use:
The below section applies to all ASP.NET load balancing configurations.
This section describes the configuration options depending on your hosting setup:
The replacement for Machine Keys in ASP.NET Core are called Data Protection. You will need to setup data protection to the same keys on all servers, without this you will end up with view state errors, validation errors and encryption/decryption errors since each server will have its own generated key.
Because Umbraco in some cases uses TempData, your setup needs to be configured with a distributed cache.
There are some logging configurations to take into account no matter what type of load balancing environment you are using.
Your staging environment should also be load balanced so that you can see any issues relating to load balancing in that environment before going to production.
You'll need to test this solution a lot before going to production. You need to ensure there are no windows security issues, etc... The best way to determine issues is have a lot of people testing this setup and ensuring all errors and warnings in your application/system logs in Windows are fixed.
Ensure to analyze logs from all servers and check for any warnings and errors.
When upgrading it is possible to run the upgrades unattended.
Here's some common questions that are asked regarding Load Balancing with Umbraco:
Question> Why do I need to have a single web instance for Umbraco admin?
TL:DR You must not load balance the Umbraco backoffice, you will end up with data integrity or corruption issues.
The reason you need a single server is because there is no way to guarantee transactional safety between servers. This is because we don't currently use database level locking, we only use application (c#) level locks to guarantee transactional data integrity which is only possible to work on one server. If you have multiple admins saving and publishing at once between servers then the order in which this data is read and written to the database absolutely must be consistent otherwise you will end up with data corruption.
Additionally, the order in which cache instructions are written to the cache instructions table is important for LB, this order is guaranteed by having a single admin server.
Question> Can my SchedulingPublisher backoffice admin server also serve front-end requests?
Yes. There are no problems with having your SchedulingPublisher backoffice admin server also serve front-end request.
However, if you wish to have different security policies for your front-end servers and your back office servers, you may choose to not do this.
No file replication is configured, deployment handles updating files on the different servers.
If the file system on your servers isn't performing any file replication then no Umbraco configuration file changes are necessary. However Media will need to be configured to use a shared location such as Blob storage or S3.
Depending on the configuration and performance of the environment's local storage you might need to consider and the .
The servers are performing file replication, updates to a file on one server, updates the corresponding file on any other servers.
If the file system on your servers is performing file replication then the Umbraco temporary folder (~/umbraco/Data/TEMP
) must be excluded from replication.
If the file system on your servers is located on shared storage you will need to configure Umbraco to locate the Umbraco temporary folder outside of the shared storage.
A common way to replicate files on Windows Server is to use [DFS](https://msdn.microsoft.com/en-us/library/windows/desktop/bb540031(v=vs.85), which is included with Windows Server.
Additional DFS resources:
There are other alternatives for file replication out there, some free and some licensed. You'll need to decide which solution is best for your environment.
When deploying Umbraco in a load balanced scenario using file replication, it is important to ensure that not all files are replicated - otherwise you will experience file locking issues. Here are the folders and files that should not be replicated:
~/umbraco/Data/TEMP/*
Achieve this by changing the value of the LuceneDirectoryFactory
setting to 'TempFileSystemDirectoryFactory' in the appsettings.json
. The downside is that if you need to view temporary files you'll have to find it in the temp files. Locating the file this way isn't always clear.
Below is shown how to do this in a Json configuration source.
~/umbraco/Logs/*
This is optional and depends on how you want your logs configured (see below)
If for some reason your file replication solution doesn't allow you to not replicate specific files folders (which it should!!) then you can use an alternative approach by using virtual directories.
The following is not the recommended setup but it is a viable alternative:
Copy the ~/umbraco/Data/TEMP
directory to each server, outside of any replication areas or to a unique folder for each server.
Create a virtual directory (not a virtual application) in the ~/umbraco/Data/
folder, and name it TEMP
. Point the virtual directory to the folder you created in step 2.
You may delete the ~/umbraco/Data/TEMP
folder from the file system - not IIS as this may delete the virtual directory - if you wish.
IIS configuration is pretty straightforward with file replication. IIS is only reading files from its own file system like a normal IIS website.
In some scenarios you have a mixture of standalone and synchronised file systems. An example of this is Azure Web Apps where the file system isn't replicated between backoffice and front end servers but is replicated between all front end servers, in this configuration you should follow the steps for synchronised file systems.
The TempFileSystemDirectoryFactory
allows Examine to store indexes directly in the environment temporary storage directory, and should be used instead of SyncTempEnvDirectoryFactory
mentioned above.
The SyncedTempFileSystemDirectoryFactory
enables Examine to sync indexes between the remote file system and the local environment temporary storage directory, the indexes will be accessed from the temporary storage directory. This setting is needed because Lucene has issues when working from a remote file share so the files need to be read/accessed locally. Any time the index is updated, this setting will ensure that both the locally created indexes and the normal indexes are written to. This will ensure that when the app is restarted or the local environment temp files are cleared out that the index files can be restored from the centrally stored index files.
You use cloud based auto-scaling appliances like
Recommended: by creating a custom IServerRegistrar
, this means the front-end servers will never be used as the SchedulingPublisher server role.
Set the UmbracoApplicationUrl
property in the
- You use cloud based auto-scaling appliances like
- Each server hosts copies of the load balanced website files and a file replication service is running to ensure that all files on all servers are up to date
- The load balanced website files are located on a centralized file share (SAN/NAS/Clustered File Server/Network Share)
ASP.NET Core supports multiple ways to share keys. Use the to find a description that fits your setup the best.
It is required to setup a distributed cache, like DistributedSqlServerCache
or an alternative provider (see for more details). The distributed cache is used by the session in your application, which is used by the default TempDataProvider in MVC.
Find steps on how to enable the feature for a load balanced setup in the article.
Alternatively store the Umbraco temporary files in the local server's 'temp' folder and set Examine to use a .
There is a specific documentation for load balancing with
If you are load balancing with make sure to check out the article we have for that specific set-up.
Once you are familiar with how flexible load balancing works, you might be interested in some .
Ensure you read the Load Balancing overview and general Azure Web Apps documentation before you begin - you will need to ensure that your ASP.NET Core & logging configurations are correct.
2 x App service plans with 1 x web app in each:
One for the backoffice (Administrative) environment
One for your scalable public-facing environment (Public)
1 x SQL server that is shared with these 2 web apps
The setup above will allow for the proper scaling of the Administrative and Public web apps.
The App Service plan with the Administrative web app should only be scaled up. The reason for this is that the web app needs to stay as a single instance.
The App Service plan with the Public web app can be scaled both out and up.
The single instance Backoffice Administrative Web App should be set to use SyncedTempFileSystemDirectoryFactory.
The multi-instance Scalable Public Web App should be set to use TempFileSystemDirectoryFactory.
When an instance of Umbraco starts up it generates some 'temporary' files on disk. In a normal IIS environment, these would be created within the folders of the Web Application. In an Azure Web App, we want these to be created in the local storage of the actual server that Azure happens to be used for the Web App. So we set this configuration setting to 'true' and the temporary files will be located in the environment temporary folder. This is required for both the performance of the website as well as to prevent file locks from occurring due to the nature of Azure Web Apps shared files system.
Host synchronization
Umbraco runs within a .NET Host.
When a host restarts, the current host 'winds down' while another host is started. This means there can be more than one live host during a restart. Restarts can occur in many scenarios including when an Azure Web App auto-transitions between hosts, you scale the instances or you utilize slot swapping.
Some file system based services in Umbraco such as the Published Cache and Lucene files can only be accessed by a single host at once. Umbraco manages this synchronization by an object called IMainDom
.
By default Umbraco v9.4 & 9.5 uses a system-wide semaphore locking mechanism. This mechanism only works on Windows systems and doesn't work with multi-instance Azure Web Apps. We need to swap it out for an alternative file system based locking mechanism by using the following appSetting. With Umbraco v10+ FileSystemMainDomLock
is the default setting.
Apply this setting to both the SCHEDULINGPUBLISHER Administrative server and the SUBSCRIBER scalable public-facing servers.
AppDomain synchronization
Each application runs inside an AppDomain which is like a subprocess within the web app process. When an ASP.Net application restarts, the current AppDomain 'winds down' while another AppDomain is started; meaning there can be more than 1 live AppDomain during a restart. Restarts can occur in many scenarios including when an Azure Web App auto transitions between hosts, you scale the instances or you utilize slot swapping.
Some file system based services in Umbraco such as the Published Cache and Lucene files can only be accessed by a single AppDomain at once. Umbraco manages this synchronization by an object called IMainDom
. By default, this uses a system-wide locking mechanism but this default mechanism doesn't work in Azure Web Apps so we need to swap it out for an alternative database locking mechanism by using the following appSetting:
Apply this setting to both the SCHEDULINGPUBLISHER Administrative server and the SUBSCRIBER scalable public-facing servers.
Create an Azure SQL database
Install Umbraco on your backoffice administrative environment and ensure to use your Azure SQL Database
Install Umbraco on your scalable public-facing environment and ensure to use your Azure SQL Database
Test: Perform some content updates on the administrative environment, ensure they work successfully in that environment, then verify that those changes appear on the scalable public-facing environment
Fix the backoffice environment to be the SCHEDULINGPUBLISHER scheduling server and the scalable public-facing environment to be SUBSCRIBERs - see Setting Explicit Server Roles
Ensure all Azure resources are in the same region to avoid connection lag.
Do not scale your backoffice administrative environment this is not supported and can cause issues.
The public-facing subscriber Azure Web Apps can be manually or automatically scaled up or down and is supported by Umbraco's load balancing.
Since you have 2 x web apps, when you deploy you will need to deploy to both places - There are various automation techniques you can use to simplify the process. That is outside the scope of this article.
This also means that you should not be editing templates or views on a live server as SchedulingPublisher and Subscriber environments do not share the same file system. Changes should be made in a development environment and then pushed to each live environment.