StreamX’s secret sauce. How the Experience Data Mesh works

StreamX uses event-driven pipelines to gather data from various sources and transform it into digital experiences ready for lightweight global delivery.

Accordingly, the StreamX Experience Data Mesh is anchored on three foundational layers: 

• unified sources - systems feeding the platform with the data using standardized messages
  

• smart distribution - platform creating experiences from incoming data in real-time
  

• lightweight delivery - services for efficient and close to the clients experiences delivery

Each one plays a crucial role in defining the platform. In the upcoming sections, we will go through problems StreamX is resolving, the design core concepts, and go through the most significant features of the platform.

Core concept:

The content management system (CMS) should be used by content authors to feed content into the platform, but the CMS itself should not be responsible for all aspects of the platform. By separating the CMS from the central point of the digital experience platform (DXP), companies can have multiple sources of content and data, integrate various APIs and services, and benefit from a loosely coupled architecture.

Traditional DXP issues improved by this approach:

• Scalability

• Integration capability

• Legacy deployment model

• Technical teams

We've reimagined the role of the CMS as a dedicated content source, focusing on its core strengths: content creation and publishing. WebSight CMS is designed to focus solely on content creation, with the content itself becoming independent once published.

Crafting experiences from any data

By transforming the CMS into a decentralized content source, we've opened the door for different types of content and data to flow through a message-oriented architecture. In StreamX , systems communicate asynchronously, allowing any system to pass data to the DXP, provided it sticks to a pre-defined message contract.

Utilizing solutions you already have

For organizations that have already invested in centralized DAM systems or have other CMSs, StreamX’s architecture offers a significant advantage, eliminating the need for costly migrations. Teams can stick with the systems they know, and the systems only need an adapter to talk to the DXP. APIs or services like PIM or CRM can be integrated into StreamX to enhance end-user experiences.

Thanks to contract-based integrations, adding new sources or replacing old ones is easier than ever. StreamX ’s loosely coupled architecture enables companies to use the right tools whenever they need them.

Core concept: Distributed event-streaming platform, offering low latency, scalability, and intelligent distribution, is the most optimal way to achieve seamless integration and enhancing the delivery of experiences.

DXP issues improved by this approach:

• Throughput

• Scalability

• Integration capability

• Not composable

• Legacy deployment model

• Technical teams

StreamX ’s biggest game changer is an event-streaming platform that we used to replace publisher-based delivery. It can manage large amounts of messages and data with low latency. The event-streaming platforms that we use made it possible to process millions of messages and gigabytes of data per second, on a mere three-node cluster. And unlike the centralized messaging systems, this platform can be scaled to hundreds of nodes.

The event-based push model

The event streaming platform is the backbone and a brain at the core of our DXP, generating, optimizing and enriching experiences with data from incoming content source messages.

With the distributed pub-sub, we decided to replace traditional request-reply with an event-based push model. StreamX stores the data from incoming systems within the persistent  logs and makes it possible to replay the events or read the historical values without the need of communicating with the originating system. 

As a result, content and data sources are no longer runtime dependencies for the experiences to be delivered. CMS, PIM, or DAM downtime won’t affect the platform availability anymore.

Smart experience generation 

Unlike in Static Site Generation (SSG), our model uses the persistent log's data to create experiences intelligently. 

For example, if new product data from the PIM arrives, StreamX pipelines generate the corresponding product page. If global price changes are made, StreamX will regenerate all product pages, leaving CMS content pages unaffected. 

With StreamX “smart pipes” we always generate as much as possible ahead of the time, and push the pre-generated experiences to the next phases.

StreamX ’s platform keeps a local persistent log, so we can plug in the extensions that can be applied to existing content without the need to re-publish everything. This means that we could, for example, add a new search module that creates search indexes reading the log's product, page, and DAM data.

Integrations via message choreography

With high-throughput and smart distribution, StreamX ’s system naturally serves as an integration point. Each broker node on the platform processes and transforms messages, eliminating single points of failure common in centralized messaging systems like JMS. 

StreamX utilizes the concept of message choreography, simplifying integration for teams. Product teams can now create their own deployments without having to rely on the integration team, as it often happens with the Enterprise Service Bus (ESB) integrations.

Core concept: Implementation of a lightweight delivery layer, powered by a smart distribution system, enables fast, cost-efficient scalability and flexible deployment options. The delivery layer can then act as an intelligent CDN, providing low latencies, immediate updates, and the ability to serve dynamic content.

DXP issues improved by this approach:

• Throughput

• Scalability

• Technical teams

The smart distribution system generates experiences with most pages pre-rendered, assets optimized, and search indexes ready to use. These experiences are stored in a log, ready for synchronization with new services. If we scale a service, for example, by adding a new HTTP server, it fetches all the available experiences automatically and subscribes to the new ones. This is a final step of a multi-stage pushing process, that ensures all services have the necessary data for efficient delivery.

Rapid and cost-efficient scaling

StreamX ’s lightweight model, automatic synchronization and a limited service responsibility, allows for rapid, cost-efficient scalability. It's flexible too; redundant services can be shut down whenever they are not needed.

Intelligent CDN

Lightweight services can be deployed across regions or even on the edge, ensuring low latencies as each service is located near end users and has local data access. This setup functions like a CDN, but with extra advantages.

In StreamX , all platform events immediately trigger changes in the service state, maintaining full resource control. Thanks to the push model, StreamX doesn’t need to be flushed like the CDN, as the delivery layer is not a cache anymore.

Another significant advantage over CDN is that experiences are available before they're consumed. The end user receives responses from the delivery layer from the very first request, and  the risk of server crashes that typically follow cache flushes gets eliminated completely.

Handling the dynamic content

StreamXis not limited to static content. With the event-based push model, we can generate experiences that in other platforms would be dynamic. An example could be a flight booking page, on which price changes are triggered by a backing algorithm using demand, interest and available seats.

In traditional DXP or in MACH  architecture, the prices would need to be fetched dynamically everytime the client visits a page. In real-time DXP, we prepare the page with actual prices before it’s accessed, and keep it consistent based on incoming events.

However, there are cases when experiences cannot be generated ahead-of-time. Such an example would be a search result page. We cannot generate search results, because we don’t know what the query is.

In such examples we create semi-products, like optimized search feeds and put it on the delivery layer for specialized and fast delivery. Requesting such a page requires a fallback to the standard request-reply model, but we make sure that it’s used only when needed.

CDN can still be used with StreamX, but it will act more like an add-on rather than a component critical for normal site operation.