Realist’s Guide to Hybrid Mesh Architecture (1): Single Source of Truth vs Democratisation

Introduction

At the crux of data mesh’s many practical issues is the inability to establish a single source of truth. In the book Data Mesh, Zhamak Dehghani calls the single source of truth a “myth” and “unachievable”, which I find an unfounded sophistry in defense of the apparent Achilles’ heel of her concept, and I vehemently disagree.

Author of the book Deciphering Data Architectures James Serra called data mesh “a solution looking for a problem”. As a data mesh skeptic myself and a staunch custodian of the single source of truth, I nevertheless find it meaningful to learn from certain principles of the data mesh concept, to improve existing centralised architecture through hybrid mesh process, while maintaining a single truth. 

I further posit that data mesh and existing hybrid mesh implementations benefit primarily the data development process, yet overlook the purpose and how data is turned into value in real world business settings. This gap creates a multitude of practical complications and confusion during implementation. To this end, I propose an alternative hybrid mesh architecture design for business value.

In this series of articles, I will:

The architecture discussed here adheres to the ELT process with data models implemented in dbt.

Data Mesh Concept & Hybrid Mesh Adoption

Six years after Dehghani proposed the data mesh concept as a solution for scaling, the idea of a pure data mesh remains highly hypothetical and unlikely to be productionised due to technological and organisational limitations.

In recent years, a hybrid mesh pattern emerged within the industry to reap the benefits of democratised domain data development; while retaining some centralised elements to lower the adoption barrier.

In a hybrid mesh architecture, each domain is empowered to develop their own data products, while all domains use the same technology stack, share the same common infrastructure, and all data live in the same central physical data lake, maintained by the central data team. (As opposed to pure data mesh which advocates domain control of different technology and infrastructure, as well as separate data lakes.)

[Figure 1: Hub-and-Spoke]

A typical strategy for hybrid mesh is the hub-and-spoke pattern (Figure 1): 

Single Source of Truth vs Democratisation

A single source of truth is not only achievable, but also a must have, for turning data into business value. We’ve seen it time and again, when domains have uncoordinated autonomy, different teams start to report vastly different numbers for crucial metrics such as revenue, under different arbitrary versions of definitions for the same metric. Data practitioners can and have long been establishing a single source of truth, which is only feasible under centralised enforcement. 

Early hybrid mesh adopters accounted for the need for conformed dimensions, canonical entities, and aligned definition of metrics. Various industry peers converged to adopt the revised Hub-and-Spoke pattern with a shared integration layer maintained by the central hub team, where data is integrated, standardised, conformed, enriched, and eventually shared among domains to form the common building blocks which domain data marts rest upon. (Figure 2)

[Figure 2: Adopted Hub-and-Spoke Pattern with a Shared Integration Layer]

This commonly adopted hub-and-spoke pattern is designed with the following principles:

Common Problems in Hybrid Mesh Implementation

The hub-and-spoke design sounds terrific on paper, until a couple months into implementation, and practical issues that the conceptual design did not account for begin to manifest themselves in front of domain developers.

Common issues encountered in practice include:

There are a few reasons for these pitfalls.

The design pattern falls under an idealised assumption that domains are distinctly independent, and there will be no overlap or cross-referencing among domains. In real world business settings this is rarely the case. 

The distinction of whether certain data should be considered shared, or owned by a domain, and by which domain, is often not crystal clear, especially when dependencies exist. Multiple domains could be involved in different stages of a single data value chain. The dependency relationships also evolve over time. Data previously consumed solely within a domain could be required by another domain later, when new business needs arise.

I have talked to a few data architects who hold the opinion that, democratised data development process comes with the inescapable tradeoff between duplication and speed. They decide to accept a certain level of compromise. “We want the domains to build the data they need. If they happen to build the same things in each domain, then so be it. At the end of the day, development needs to move forward.”

Others, like me, who are more adamant on the single source of truth, heed another approach.

Practical Solutions for Hybrid Mesh Implementation

Let me be upfront on this: The practical solution to the mesh-y domain problems is almost always – moving upstream. In other words, reverting back to centralisation, and moving away from mesh.

The rules are simple: 

[Figure 3: Hybrid Mesh Hub-and-Spoke Architecture Practical Solution]

Figure 3 illustrates the eventual architecture in practical implementation. All data which is shared and reused across domains resides in the Central Integration layer, establishing a single source of truth, and reconciling the dependency complications. 

In Part 2 of this article, I will explain in detail with examples what each layer contains, and the dependencies among the layers.

Alternative Hybrid Mesh Architecture Design for Business Value

The further downstream we move on the data value chain, the more dependencies interlock the business insights. Consider for example the need to aggregate various facts, to construct profiles or segments on customers or products, and automate actions based on the profiles. 

To meet real world business requirements, data from various domains often need to be combined together at the end of the lineage, where data is turned into business value. The existing hub-and-spoke solution primarily benefits the data development process, yet is not optimised for creating value from data. 

Note the distance between the simplicity of the hub-and-spoke conceptual design in Figure 2, and the actual complexity during implementation illustrated by Figure 3. 

To better accommodate for real world data modelling needs for creating business value, and to reduce complexity in the architecture in practice, I propose an alternative design to the hub & spoke pattern: the Hybrid Mesh Constellation Architecture.

[Figure 4: Hybrid Mesh Constellation Design Pattern]

The Constellation design (Figure 4) observes similar rules as the practical hub-and-spoke solution. A central integration layer is established upstream, as the basis for domain development, with the addition of another centralised layer at the end, where data from the whole organisation is aggregated to generate holistic insights for the business.

[Figure 5: Hybrid Mesh Constellation Architecture in Practice]

The Constellation Architecture in practice (Figure 5) follows these principles:

In Part 2 of this article, I will elaborate on the different layers of this architecture, and outline an actionable decision flow.

Hybrid Mesh as Governance Architecture

By now, I’m sure you’d have noticed the elephant in the room: By moving data to a centralised location (in solving the hybrid mesh implementation problems), our architecture reverts towards the traditional centralised architecture, and away from mesh. So why did we choose to implement a mesh pattern in the first place?

I apply the mesh principles not in the data models’ provenance, but in their properties.

Think of Hybrid Mesh less in terms of residence architecture, but more as governance architecture. I advocate for centralised data residence with democratised execution.

I will detail on the complete governance architecture in Part 2, including practical guidelines on “who does what and when”.