Data mesh: Transforming data from byproduct to product

One of the four principles of data mesh is data as a product.

The core tenet of this principle is a shift in mindset from data as a byproduct of transactional systems and processes to data purposefully designed and packaged as a “product” for an analytical need.

This shift in mindset is facilitated by applying product management practices to the design of data products, including defining the product vision and strategy, creating the development roadmap, and ongoing management of quality and usability.

Activities in the product management process include:

• Understanding the user: The first step in the product management process is to understand the user. This includes understanding the user's needs, wants, and pain points.
  
  
• Scoping the problem: Once you understand the user, you need to define the scope of the problem that your product will solve. What is the user trying to achieve? What are the obstacles that are preventing them from achieving their goals?
  
  
• Setting goals: What metrics will be used to measure product success? How can progress against the metrics be tracked and communicated?
  
  
• Brainstorming solutions: Once you have defined the problem, you can start brainstorming solutions. What are the different ways that you can solve the user's issue?
  
  
• Prototyping: Once you have some ideas for solutions, you can start prototyping. A prototype is a rough, working model of your product. It doesn't have to be perfect, but it should be enough to give you a sense of how the product will work.
  
  
• Testing: Once you have a prototype, you must test it with users. This will help you identify any problems with the product and ensure it meets the user's needs.
  
  
• Iterating: Once you have tested your product, you need to iterate. This means making changes to the product based on the feedback that you received from users.

The shift in mindset to data products requires new skills and disciplines that your current data and analytics teams might not have.

Don’t assume data stewards, engineers, and scientists should be repurposed as data product managers. Define the requirements first, and then determine who will be a good fit based on them. You may need to look outside traditional data and analytics roles to find the right people.

Some things to consider include:

As mentioned in our article, Data Mesh: Understanding Decentralized Domain Ownership, with data mesh, responsibility and accountability for data are distributed to domain teams that best understand the business needs and context.

This includes creating and managing data products. While local domain autonomy determines data product characteristics, such as business entities, attributes, and hierarchies, domain teams are also responsible for ensuring the interoperability and reusability of data products across domains.

• Discoverability: Data products should be easy to find and explore. This can be done by providing a central catalog or registry of data products, as well as by making data products discoverable through search.
  
  
• Addressability: Each data product should have a unique, persistent address that can be used to access it programmatically or manually. This address should be consistent even as the data product evolves.
  
  
• Understandability: Data products should be easy to understand. This means providing clear documentation about the data product's semantics, syntax, and schema. It also means giving sample datasets and example code to help users get started.
  
  
• Trustworthiness & truthfulness: Data products should be trustworthy and truthful. This means providing information about the data product's quality, timeliness, completeness, and accuracy. It also means providing data provenance and lineage information.
  
  
• Native accessibility: Data products should be accessible to a wide range of users, with different needs and expectations. To do this, data products should be made available in multiple formats, so that users can access them using their preferred tools. For example, a data product could be made available as a spreadsheet, a SQL database, or a REST API.
  
  
• Interoperability: Data products should be easily combined with other data products. This allows users to create more complex and robust analyses. To make data products interoperable, they should conform to the minimum standards defined as part of your governance activities.
  
  
• Valuable on its own: Data products should be useful to users, even if they are not combined with other data products. This means that data products should be well-curated and contain high-quality data. Additionally, data products should be well-documented so that users can understand their contents and how to use them.
  
  
• Secure: Data products should be secure so that users can be confident that their data is protected. This means that access should be controlled, and there are clear policies for compliant use. Additionally, data products should be regularly monitored for security vulnerabilities.

Conclusion

Data products are the primary deliverables of the data mesh, enabling organizations to make business decisions faster and quickly adapt to changing business conditions.

The change in mindset from data as a byproduct to data as a product requires more rigor and discipline in curating and consuming data for analytical needs.