Umbraco Engage
CMSCloudHeartcoreDXP
13.latest (LTS)
13.latest (LTS)
  • Umbraco Engage Documentation
  • Release Notes
  • Support
  • Installation
    • System Requirements
    • Installation
    • Licensing
    • Verify your Engage installation
  • Upgrading
    • Upgrade Umbraco Engage
    • Version specific Upgrade Notes
    • Migrate from uMarketingSuite
  • Getting Started
    • Getting Started
    • For Marketers and Editors
      • Cockpit
      • Marketing Resources
    • For Developers
      • Infrastructure sizing
      • Load Balancing and CM/CD Environments
      • Content Delivery Network recommendations
      • Cockpit
      • Content Security Policy nonce configuration
      • Troubleshooting installations
  • Marketers and Editors
    • Introduction
      • The Umbraco Engage Section
      • Content Apps
      • The Umbraco Engage Cookie
    • Analytics
      • What is measured by default
      • Client-side Events
      • Types Of Clients
      • Campaigns
      • Device Type
      • Location
      • Referral Traffic
      • Forms
      • Videos
      • Scroll Heatmap
      • Google Analytics vs Umbraco Engage
      • Search Terms
    • A/B Testing
      • What is A/B testing
      • Types of A/B Tests
        • Single-page A/B Test
        • Multiple Pages Test
        • Document Type Test
        • Split URL Test
      • Setting up the A/B Test
      • Previewing an A/B Test
      • Monitor the A/B Test
      • A/B Test Distribution Algorithm
      • Front end Rendering
      • Finish an A/B Test
    • Personalization
      • Creating a Segment
      • Setting up Personalization
      • Cockpit Insights
      • Implicit and Explicit Personalization
        • Setting up the customer journey
        • Personas
        • Implicit Personalization scoring explained
        • Content Scoring
        • Campaign Scoring
        • Referral Scoring
    • Profiling
      • Profile detail
      • External profile data
    • Reporting
    • Settings
      • Goals
      • IP Filtering
      • Configuration
      • Permissions
  • Developers
    • Introduction
      • Dataflow Pipeline
        • Data Collection
        • Data Storage
        • Data Parsing
        • Reporting
      • The Umbraco Engage Cookie
        • Module Permissions
      • Performance
    • Analytics
      • Request tracking
      • Bot detection
      • Capture location data
      • Extending forms
      • Video tracking
      • Scroll Heatmap
      • Client-side events
        • Additional measurements with analytics scripts
        • Bridging Library for Google Analytics
        • Bridging Library for Google Tag Manager
        • Google Analytics Blocker Detection
        • Create your own events
      • Extending Analytics
        • Getting the Correct IP Address
        • Sending data to the GTM Datalayer
    • A/B testing
      • Retrieving A/B test variants in C#
    • Personalization
      • Implement your own segment parameters
      • Retrieve segment information from code
      • Add custom scoring
    • Profiling
      • External Profile Data
    • Reporting
    • Settings
      • Custom goals scoring
      • Configuration
    • Headless
      • Using the Engage API
      • Headless Example
  • Security and Privacy
    • Security and privacy
    • Retention periods of data
    • Anonymization
    • GDPR & EU regulation
      • How to become GDPR compliant using cookiebot
    • How it works
  • Tutorials
    • Overview
    • How to Get Started with Personalization
    • How to Create a Persona
    • Create a Personalized Popup in 5 minutes
    • How to set up an A/B Test
    • Marketing Resources
      • Generic Topbar Template
      • Generic Popup Template
      • Generic Exit Intent Popup Template
Powered by GitBook
On this page
  • Flushing the memory
  • Data storage

Was this helpful?

Edit on GitHub
Export as PDF
  1. Developers
  2. Introduction
  3. Dataflow Pipeline

Data Storage

Information about Data Storage and how to work with and troubleshoot it in Umbraco Engage.

PreviousData CollectionNextData Parsing

Last updated 6 months ago

Was this helpful?

When the it is temporarily stored in memory. At some point, a threshold is reached and all data is stored in the database.

Flushing the memory

Two thresholds can be set and reached which will trigger the storage of data. If one of these two is reached the data will be stored in the database.

  • The first threshold is the 'FlushRateInRecords'.

    • When this number of records is in memory the data will be stored in the database. An example could be if you set it to 100, the data will be permanently stored after 100 page visits.

  • The second threshold is the FlushIntervalInSeconds.

    • After this number of seconds, the data will be sent to the database. If you set it to 30 seconds, for example, every 30 seconds the data will be sent to the database. No matter how many records there are in memory.

Both settings can be set in the file of Umbraco Engage.

The higher the value set for these thresholds, the more memory Umbraco Engage uses on your web server(s) and less of your database connection. Please be aware the memory impact is low because there is not a lot of complex data stored.

The lower the value you set, the less memory Umbraco Engage uses on your web server(s), and the more database calls are made.

Data storage

The data will be stored as quickly as possible to minimize the needed resources. For this reason, the the data collected from client-side events will be stored in so-called raw tables in a non-normalized. This data will be processed in of the data flow.

The data collected from clientside events is stored in the table umbracoEngageAnalyticsRawClientSideData.

When the data is stored in these tables the columns processingStarted, processingFinished, processingMachine, and processingFailed are empty. They will be filled in the parsing step.

data is collected
configuration
the next step