Following up on Cerys’ exploration of building data capabilities last week, I want to discuss the potential that a Data Mesh approach holds for businesses to achieve this, compared to more centralised data warehouses and data lakes (or even data lakehouses!). Perhaps its most valuable and profitable characteristic is its focus on how organisations and people should interact with and own data.
Spoiler alert: it’s not like a squirrel hoarding nuts for winter.
Data-Driven us crazy
I used to work in a public sector organisation, where my job involved questioning our processes and ways of working, and working with the wider team to improve them – from simple timing adjustments to changes in our CRM and policies.
I worked closely with someone who was an Excel wizard. It was a partnership made in heaven. I looked at the datasets and reports made available to us and asked questions like ‘what if?’, ‘but how?’, and ‘surely we can?’, while my partner would translate these musings into formulas, pivot tables and VLOOKUPs as I was talking. But there is one little phrase in the previous sentence that complicated things:
“datasets and reports made available to us”
Why did we have to be given the data? As is common, the organisation was too siloed, with a central IT department that also housed a relatively young Business Intelligence (BI) Team that worked on data dashboards and analytics for all 6 main departments. The desire to be ‘data-driven’ was still a work in progress, so it was all hands on deck. This centralised configuration led to multiple problems:
- The wrong person determining what information is relevant: The reports that were built, both historically and in the form of data dashboards, were built without much involvement from domain experts, leading to incomplete and ambiguous reporting.
- Hoarding and controlling resources led to huge delays: When reports were incomplete or wrong, change requests went into a long queue of requests from all departments. And a lack of domain knowledge led to poor prioritisation of requests, costing departments time, money and KPIs.
- Increased resentment towards IT: Internal customers lost trust in and resented IT and the BI team because of delays and wasted time. In the end, many people stopped caring about accurate information, or developed elaborate, time-expensive work-arounds.
As two people committed to helping our department deliver, our life in a supposedly data-driven organisation was sub-optimal. We wasted a lot of time and energy on ‘best efforts’ guesstimates to inform leadership – it was all we could do, but very frustrating since we knew there was a better way.
The solution?
Decentralise data analytics and engineering into local departmental data domains.
This not only shortens communication, but also provide BI experts with domain context, and gives business leaders and teams direct access to data expertise. This is one of the core cultural principles of a Data Mesh Approach. In an organisation that has implemented a Data Mesh Approach, as Zhamak Deghani in 2019 coined the term, data is seen as a product owned and managed by the domains that produce them, and is often intimately linked to how the data is consumed within these domains. Of course, they are also accessible to everyone across the organisation. If this had been the case for my Excel-partner-in-crime and I, our lives would have been a lot easier.
Source: https://www.expressanalytics.com/blog/what-is-data-mesh-its-architecture-and-benefits/
https://www.expressanalytics.com/blog/what-is-data-mesh-its-architecture-and-benefits/
As someone who is very passionate about Data Quality and Data Literacy in organisations, I am convinced that bringing BI developers and data analytics out of the Ivory Tower and directly into the domains helps improve overall data quality and also breach the data language barrier that is holding so many people back from delivering on their potential, as my colleague Cerys wrote about recently. This can improve data literacy, trust in data, contribute to more accuracy in decision-making, improve relationships between domains, and in the end, improve our businesses.
At least in terms of data management, perhaps Joe Peppard was right when he wrote ‘It’s time to get rid of the IT department’…?
Data Mix and Mesh: the end of IT Ivory Towers?
Of course, it is not as simple as just embedding data engineers into departments. There are, as Adam Bellemare writes, more fundamental organisational changes you have to make, including a new approach to data governance and security, and infrastructure will still cost a lot of money. Data Mesh might not be the right fit for every organisation, for reasons of cost, scale or capabilities. But even in the case I shared above, a partially decentralised approach to data analytics achieved by placing BI people within the departments/domain would have set us up for success rather than frustration and failure. It is in the interdependency of the technical with the socio-cultural experience and ownership of data that many of the frustrations from my story could have been prevented or alleviated. Data, in organisations where IT and data management are highly centralised, becomes quite abstract – measured in aggregate and embedded in PowerPoints to show how well teams and departments have done. The words and numbers on the screens and reports are just that. There is often no perceived or tangible connection between what people enter or retrieve from a system, because what the words and numbers mean, where they come from, and where they are going, are all obscured by a combination of techie language, boring reports and lack of context or connection to reality.
Photo by Bob Brewer on Unsplash