How hybrid integration platforms can help to enable Data Mesh
Traditional approach of data management in the enterprises is not enough to support the dynamic of the market and the context of omnichannel, ecosystems of partners and modern hybrid architectures. Data Mesh is a new concept to help this challenge with data and Hybrid Integration platforms (HIP) come to help this kind of implementation.
This article is my first impression of how HIP can help and we can start talking about the 2 main concepts, Data Mesh and Hybrid Integration Platforms.
Data Mesh was first defined by Zhamak Dehghani, a ThoughtWorks consultant and the original architect of the term, a data mesh is a type of data platform architecture that considers the omnipresence of data in the enterprise by leveraging a domain-oriented, self-serve design.
Data mesh uses the concept of Domain where you change your view of the monolithic approach of Big Data and change to Domain of Data, and this silos control the data providing it to others.
Hybrid integration platforms (HIP) are a set of technologies based on premises or cloud, in some cases pure native cloud to enable interoperability of systems becoming in charge of pipelines of data flow. HIP works in a consumer provider model integrating technologies.
The data in each domain can use any kind of technology to store and format, and must be in charge to prepare the data to be consumed by other domains. Following the example of Zhamak you can have the domain of music players using one kind of storage and the consumer can be the domain of top artists using another kind of storage and data format.
From Zhamak article: This implies that we may duplicate data in different domains as we transform them into a shape that is suitable for that particular domain, e.g. a time series play event to related artists graph.
Considering this, we need to rethink the model of data transfer and transformation in general and HIP can help here.
To push and pull data across all domains, a platform based on HIP can help to talk any language and format to collect, use techniques and tools to transform and deliver the data in the correct format.
Not more using a central monolithic approach (Big Data) but directly to domains and enabling in a transition step the both approaches, sending data to Big Data and sending to Domains.
From Zhamak article: Note that the source aligned domain datasets must be separated from the internal source systems’ datasets. The nature of the domain datasets is very different from the internal data that the operational systems use to do their job.
Considering all the points of this article and the vision of Zhamak article, this is the big picture with HIP:
In conclusion, Data Mesh Architecture and framework plus Hybrid Integration Platforms is about empowering and enabling business domains to be involved, responsible and accountable in the data creation process. A business domain is responsible for the definition of data, the quality of data and the creation of data. Business domains produce and sell actionable data. In order to achieve the goal of Data Mesh and get the outcome from it, the HIP are the enablers.
