How to extend Umbraco Deploy to synchronize custom data
Umbraco Deploy supports the deployment of CMS schema information and definitions from the HQ's Forms package, along with managed content and media. Additionally, it can be extended by package or custom solution developers. This allows the deployment of custom data, such as that stored in your own database tables.
As a package or solution developer, you can hook into the disk-based serialization and deployment. It is similar to that used for Umbraco Document Types and Data Types. It's also possible to provide the ability for editors to deploy custom data via the Backoffice. In the same way that Umbraco content and media can be queued for transfer and restored.
Entities are what you may be looking to transfer between two websites using Deploy. Within Umbraco, they are the Document Types, Data Type, documents etc. In a custom solution or a package, there may be representations of some other data that's being stored separately from Umbraco content. These can still be managed in the Backoffice using custom trees and editors.
For the purposes of subsequent code samples, we'll consider an example entity as a Plain Old Class Object (POCO) with a few properties.
The entity has no dependency on Umbraco or Umbraco Deploy; it can be constructed and managed however makes sense for the package or solution. The only requirement is that it has an ID that will be consistent across the environments (normally a Guid) and a name.
Every entity Deploy works with, whether from Umbraco core or custom data, needs to have an artifact representation. You can consider an artifact as a container capable of knowing everything there is to know about a particular entity is defined. They are used as a transport object for communicating between Deploy environments.
They can also be serialized to JSON representations. These are visible in the .uda files seen on disk in the /data/revisions/
folder for schema transfers. It is also used when transferring content between different environments over HTTP requests.
Artifact classes must inherit from DeployArtifactBase
.
The following example shows an artifact representing the entity and it's single property for transfer:
In most cases the default settings Umbraco Deploy uses for serialization will be appropriate. For example, it ensures that culture specific values such as dates and decimal numbers are rendered using an invariant culture. This ensures that any differences in regional settings between source and destination servers are not a concern.
If you do need more control, attributes can be applied to the artifact properties.
For example, to ensure a decimal value is serialized to a consistent number of decimal places you can use the following. RoundingDecimalJsonConverter
is found in the Umbraco.Deploy.Serialization
namespace
Service connectors are responsible for knowing how to handle the mapping between artifacts and entities. They know how to gather all the data required for the type of entity they correspond to, including figuring out what dependencies are needed. For example, in Umbraco, how a Document Type depends on a Data Type. They are responsible for packaging an entity as an artifact and for extracting an entity from an artifact and persisting it in a destination site.
Service connectors inherit from ServiceConnectorBase
and are constructed with the artifact and entity as generic type arguments.
The class is decorated with a UdiDefinition
via which the name of the entity type is provided. This needs to be unique across all entities so it's likely worth prefixing with something specific to your package or application.
The following example shows a service connector, responsible for handling the artifact shown above and it's related entity. There are no dependencies to consider here. More complex examples involve collating the dependencies and potentially handling extraction in more than one pass to ensure updates are made in the correct order.
An illustrative data service is provided via dependency injection. This will be whatever is appropriate for to use for Create, Read, Update and Delete (CRUD) operations around reading and writing of entities.
In Deploy 9.5/10.1, to improve performance on deploy operations, we introduced a cache. This change required the addition of new methods to interfaces, allowing the passing in of a cache parameter. In order to introduce this without breaking changes, we created some new interfaces and base classes.
In the example below, if instead we inherited from ServiceConnectorBase2
, which has a type parameter of IServiceConnector2
, we would be able to implement IArtifact? IServiceConnector2.GetArtifact(Udi udi, IContextCache contextCache)
. This would allow the connector to read and write to the cache and remove the use of the obsolete methods.
There's no harm in what is listed below though. It's only that the connectors won't be able to use the cache for any look-ups that are repeated in deploy operations. The obsolete methods won't be removed until Deploy 11. In that version we plan to return back to the original interface and class names. We also plan to introduce the new method overloads which will be a documented breaking change.
It's also necessary to provide an extension method to generate the appropriate identifier:
Umbraco entities often have dependencies on one another, this may also be the case for any custom data you are looking to deploy. If so, you can add the necessary logic to your service connector to ensure dependencies are added. This will ensure Umbraco Deploy also ensures the appropriate dependencies are in place before initiating a transfer.
If the dependent entity is also deployable, it will be included in the transfer. Or if not, the deployment will be blocked and the reason presented to the user.
In the following illustrative example, if deploying a representation of a "Person", we ensure their "Department" dependency is added. This will indicate that it must exist to allow the transfer. We can also use ArtifactDependencyMode.Match
to ensure the dependent entity not only exists but also matches in all properties.
As well as dependencies at the level of entities, we can also have dependencies in the property values as well. In Umbraco, an example of this is the multi-node tree picker property editor. It contains references to other content items, that should also be deployed along with the content that hosts the property itself.
Value connectors are used to track these dependencies and can also be used to transform property data as it is moved between environments.
The following illustrative example considers a property editor that stores the integer ID of an media item. The integer ID of a media item is not consistent between environments, so we'll need to transform it. And we also want to ensure that the related media item itself is transferred as well as the integer ID reference.
With the artifact and connectors in place, the final step necessary is to register your entity for deployment.
Connectors do not need to be registered. The fact that they inherit from particular interfaces known to Umbraco Deploy is enough to ensure that they will be used.
If custom entity types are introduced that will be handled by Umbraco Deploy, they need to be registered with Umbraco to parse the UDI references.
This is done via the following code, which can be triggered from a Umbraco component or an UmbracoApplicationStartingNotification
handler.
To deploy the entity as schema, via disk based representations held in .uda
files, it's necessary to register the entity with the disk entity service. This is done in a component, where events are used to trigger a serialization of the entity to disk whenever one of them is saved.
In Umbraco Deploy 9.4 a schema comparison feature was added to the dashboard available under Settings > Deploy. This lists the Deploy managed entities held in Umbraco and shows a comparison with the data held in the .uda
files on disk.
All core Umbraco entities, such as Document Types and Data Types, will be shown.
To include entities from plugins, they need to be registered using a method overload as shown above, that allows to provide additional detail, e.g.:
The parameters are as follows:
The system name of the entity type (as used in the UdiDefinition
attribute).
A human readable, pluralized name for display in the schema comparison dashboard user interface.
A flag indicating whether the entity is an Umbraco one, which should be set to false
.
A function that returns all entities of the type installed in Umbraco, mapped to an object exposing the Udi and name of the entity.
If the optimal deployment workflow for your entity is to have editors control the deployment operations, the transfer entity service should be used. This would be instead of registering with the disk entity service. The process is similar, but a bit more involved. There's a need to also register details of the tree being used for editing the entities. In more complex cases, we also need to be able to handle the situation where multiple entity types are managed within a single tree.
An introduction to this feature can be found in the second half of this recorded session from Codegarden 2021.
There's also a code sample, demonstrated in the video linked above, available here.
The following code shows the registration of an entity for Backoffice deployment, where we have the simplest case of a single tree for the entity.
The RegisterTransferEntityType
method on the ITransferEntityService
takes the following parameters:
The name of the entity type, as configured in the UdiDefinition
attribute associated with your custom service connector.
A pluralized, human-readable name of the entity, which is used in the transfer queue visual presentation to users.
A set of options, allowing configuration of whether different deploy operations like queue for transfer and partial restore are made available from the tree menu for the entity.
A value indicating whether the entity is an Umbraco entity, queryable via the IEntityService
. For custom solutions and packages, the value to use here is always false
.
The alias of the tree used for creating and editing the entity data.
We then have three functions, which are used to determine if the registered entity matches a specific node and tree alias. For trees managing single entities as we have here, we know that all nodes related to that tree alias will be for the registered entity. And that each node Id is the Guid identifier for the entity. Hence we can use the following function definitions:
Return true
for all route paths.
Return true
for all node Ids.
Return true
and parse the Guid identity value from the provided string.
For more complex cases we need the means to distinguish between entities. An example could be when a tree manages more than one entity type. Here we would need to identify whether the entity is being referenced by a particular route path or node ID. A common way to handle this is to prefix the GUID identifier with a different value for each entity. It can then be used to determine to which entity the value refers.
For example, as shown in the linked sample and video, we have entities for "Team" and "Rider", both managed in the same tree. When rendering the tree, a prefix of "team-" or "rider-" is added to the Guid identifier for the team or rider respectively. We then register the following functions, firstly for the team entity registration:
And then for the rider:
If access to services is required when parsing the entity ID, where the HttpContext
is provided as a parameter, a service can be retrieved. For example:
The remoteTree
optional parameter adds support for plugins to implement Deploy's "partial restore" feature. This gives the editor the option to select an item to restore, from a tree picker displaying details from a remote environment. The parameter is of type DeployRegisteredEntityTypeDetail.RemoteTreeDetail
that defines three pieces of information:
A function responsible for returning a level of a tree.
The name of the entity (or entities) that can be restored from the remote tree.
The remote tree alias.
An example function that returns a level of a remote tree may look like this:
Finally, the entitiesGetter
parameter allows you to pass a function that will return all entities for the registered type. This is necessary to allow support for the "set signatures" operation available via the backoffice settings dashboard. By providing a function that returns the collection of entities, the triggered operation will be able to prepare the signature for each one.
To complete the setup for partial restore support, an external tree controller needs to be added, attributed to match the registered tree alias. Using a base class available in Umbraco.Deploy.Forms.Tree
, this can look like the following:
Most features that are available for the deployment of Umbraco entities will also be accessible to entities defined in custom solutions or packages. The "Transfer Now" option available from "Save and Publish" or "Save" button for content/media is only available to custom entities if requested client-side.
You would do this in the custom AngularJS controller responsible for handling the edit operations on your entity. Inject the pluginEntityService
and calling the addInstantDeployButton
function as shown in the following stripped down sample (for the full code, see the sample data repository linked above):
Umbraco Deploy improves the efficiency of transfers by caching signatures of each artifacts in the database for each environment. The signature is a string based, hashed representation of the serialized artifact. When an update is made to an entity, this signature value should be refreshed.
Hooking this up can be achieved by applying code similar to the following, extending the ExampleDataDeployComponent
shown above.