Optimizing Real-Time Data Integration from Multiple Sources


This article explores  how WebSight DXP can introduce real-time data management from multiple sources for a large retailer, significantly enhancing the user experience and operational efficiency. The implementation can lead to faster product updates along with improved brand trust and sales. The result: seamless integration of data from diverse sources, improved customer experience and streamlined operational processes.

The Business Landscape

For large retailers that offer hundreds of thousands of products on its website, one of the more significant IT challenges is ensuring product information stays up to date across the entire website, including product pages and  search functionality. This is a factor that significantly affects conversion rates and, ultimately, company’s financial results.

Challenges in Daily Updates

Key elements of the overall store's architecture come into play in this process:

  • The Digital Experience Platform (DXP), along with a Content Management System (CMS), used for creating and designing web pages.

  • Product Information Management (PIM) system for updating product images and descriptions.

  • A search engine that customers expect to deliver accurate results for different keywords

  • A price management system

In many retail scenarios, the DXP platform bears the largest responsibility for the final user experience and updating information on the site, hence it is often known as the system "holding the glass" or maintaining the final display of data.

In practice, this means changes introduced in the PIM system, integrated with the CMS, will only appear on the site once the DXP platform with CMS regenerates the page. 

With a high number of products, this can take hours. Hence, engineers tasked with maintaining e-commerce platforms usually schedule PIM updates for nighttime, when the site traffic is at its lowest.

The Problem

Despite these measures, a key problem persists: the delay for the DXP to display changes also postpones synchronization with other data sources. This often leads to issues such as inaccurate or outdated search engine results, mismatches between the price on the site and in the cart, or complications with applying discounts.

All these challenges can result in significant conversion and revenue losses for online stores. Studies have shown that more than half of consumers abandon their purchase if something about the store's operation frustrates them - source.

Strategy Shift = Improved Outcomes

WebSight DXP  introduces a key strategic shift - building user experiences around events that occur in the ecosystem, rather than attempting to align systems. With an event-driven DXP, systems produce events that are detected and consumed by services responsible for intelligent page generation.

Instead of having all systems communicate directly with each other, the integration with an event-driven DXP allows them to feed events to the processing pipelines, which then take up the task of generating, composing and optimizing the final page.

Ahead-of-time page generation

Let's explore what this means for data updates in an e-store catalog. 

In a scenario like promotion or product page template change, the entire catalog of 300 000 of products requires updates. In a traditional request-reply model, implementing those changes would be a time-consuming process. First, they would need to be transferred from one monolith (the author) to multiple monoliths (publishers). Then, the publisher would get ready to render the page, that would then get stored HTTP cache. To give everything enough time to process, we would need to schedule the batch to update for 8 to 12 hours during the night. 

In an event-driven environment, every change applied to the Product Information Management (PIM) system triggers the generation of the page 'ahead of time,'. The page is then immediately placed close to the user. This means the final page is available for the users even before they make any request.

With WebSight’s high-throughput pipelines, we can accomplish the outcome that takes the traditional DXP 8 to 12 hours, within approximately 10 minutes. Although this timeframe could be further reduced, keeping it around 10 minutes for the 300 000 products proved to be the most cost-effective approach in terms of cloud infrastructure.

Moreover, with this updates model, we reduce the need for batch updates altogether,  and reserve them for “special occasions” like global template change. If we react to a product data change with a product page update whenever product data is modified in the PIM, we can achieve continuous updates flow, and reduce the reliance on bulk processing.

Unified search

In many e-commerce scenarios, the search function that the customers use to look for products is actually two or more separate services.

If there are multiple sources of search data, we need to find a way to provide a unified customer experience. One solution is to create a search index per source and compose it on the frontend. Alternatively, the same composition can be done by the backend service.

This is typically necessary when the CMS stores marketing data, while the PIM provides more specific information such as product codes and parameters. The CMS may store a portion of the data from the PIM if the CMS "holds the glass."

However, integration like that can be tricky. It's difficult to unify the search view from multiple sources, and it can result with:

  • problems with ranking

  • problems with indexing all the data

  • problems with matching search results from multiple sources

The logic implementation is distributed across multiple places, including the PIM, CMS for feeding the indexes, the indexes themselves, and the frontend application. There are at least 4 services to orchestrate, which adds complexity to the system.

WebSight resolves this issue by taking all the necessary data directly from CMS and the PIM. It then feeds it  into a search index. Whenever new elements are added, they're simply incorporated into the pipeline, which then automatically updates the search interface. This streamlines the user experience and ensures that all product data is easily accessible.

The responsibility for maintaining data consistency no longer rests on Single Source of Truth. In WebSight event-driven DXP approach, data sources are integrated and synchronized in real time, and data is searchable across all systems.

Business Impact

The online customers now see consistent product information, regardless of the time or other factors. This consistency enhances brand trust, customer loyalty and leads to increased sales.

By using WebSight DXP, the retail company can maintain an accurate, up-to-date product catalog with consistent user experience. This approach can ultimately lead to improved conversion and increased sales. 

Let's sum up the  main benefits of embedding WebSight event-driven DXP into a web store architecture:

  1. Real-time Data Processing: The ability to process data in real-time ensures continuous up-to-date product information. This means users always have access to the most current data.

  2. Improved Efficiency: The time to update the product catalog has been drastically reduced from hours to mere minutes, increasing operational efficiency

  3. Enhanced User Experience: Thanks to a unified view of data from various sources, customers now see accurate and consistent product information regardless of the time of day. This enhances the customer experience, boosts brand trust, and drives sales

  4. Scalability and Flexibility: WebSight's architecture is designed for scalability and flexibility, enabling the integration of new technologies and strategies efficiently. This capability can help the retailer adapt quickly to changing market conditions and user needs

More possibilities for the future

With all content passing through the same pipeline, the process of creating web pages, for both front-end applications and back-end systems, is simplified. This not only means faster and more efficient operations, but also facilitates quick marketing campaign deployment, A/B testing, and responsive adjustments to any emerging user needs.