Enabling AI Through the Data Mesh

The Department of Defense (DOD) is looking to expand its data infrastructure with modern data concepts, key to creating decision advantage. Booz Allen is helping DOD pioneer a data mesh architecture to enable a decentralized, domain-oriented model to ingest, process, govern, scale, secure, and manage the lifecycle of data assets.

A data mesh concept is defined as a platform that is intentionally designed as a distributed data architecture while maintaining centralized governance and standardized operations. Data mesh is a decentralization of analytical data design that is analogous to domain autonomy in microservices architectures.

A data mesh architecture is designed to enable easy exchange of data between domains and ownership. Currently, since data is inconsistently processed, users cannot easily discover, access, and use available information. As a result, many data products created by DOD are limited in scope, utility, and impact. A data mesh provides the advantage of a common core, allowing data teams to perform functions at scale across an integrated organization.

Here are the four main principles of a data mesh:

1. Data as a product: Data assets are built proactively for business and mission demands to provide value.
2. Domain ownership: Domains are decentralized, with responsibility distributed among the people closest to the data.
3. Self-serve data, infrastructure as a platform: Abstract infrastructure complexity and provision resources to enable self-service data access and data management.
4. Federated computational governance: Interoperability is ensured through a global standard, with automated execution of platform decision to enable self-service.

Agencies must take a decentralized, sociotechnical approach to managing and accessing data at scale—taking organizational culture and users’ interaction with technology into account. The DOD Data Strategy describes the goals toward achieving data-centricity as VAULTIS: visible, accessible, understandable, linked, trustworthy, interoperable, and secure—all aligning to the data mesh concept.

To enable data mesh capabilities, DOD must evolve to make code, data, and metadata accessible to other data domains. This starts by evolving infrastructure, enabling teams to make timely, accessible data products having security policies that enforce compliance.

Like any other modernization component, it is important to map data to business objectives. For efficacy, data should be organized based on how it is used rather than where it is stored. Users should have a unified experience and be able to quickly access distributed information from approved DOD and open-source tools with assigned metadata.

In an operational data mesh architecture, data is stored across the agency in respective domains. Data domains support a federated data architecure, maintain centralized governance, and standardize operations. This decentralized framework enables organizations to scale on demand without compromising performance. DOD can ensure its enterprise framework is structured to manage the demand of data across multiple domains by following these fundamentals:

• Self-service: Enables easy access to data and reduces time to a minimal viable product (MVP). Federated computational governance must be prioritized to not only identify policies but to enforce agency regulations and standards. Infrastructure as code (IaC) is often leveraged to automate this process across user groups, organizations, and affiliated stakeholders.
• Feedback loops: Modernization is an iterative process. Building a community of practice for data products and iterating the architecture based on user feedback allows information to scale as mission and business requirements change.
• Metric utilization: Understanding how data is being used and leveraging user analytics enables organizations across the agency to deploy new products. These timely data products deliver quality solutions, increase value, and support mission and business growth. 

Additionally, data quality must be comprehensive, shareable, secure, and built for use as a mission product rather than for a specific user group. Considered as a sphere of knowledge, a data domain is established organically and aligns to business or mission organizations closest to the data, source, or primary consumers. These domains curate and serve data products to meet user-specific business and mission objectives. Data products must be understandable, high quality, secure, and easily accessible through a well-defined interface.

Measuring the value of data products involves measuring qualitative and quantitative user product performance, as well as evaluating the process used to create the product. Collected metrics and key performance indicators (KPI) enable organizations to evolve the delivery of products and relevant information.

Organizations are using metrics and KPIs to address performance as data complexity continues to grow and surpass both network ability and mission requirements. Agencies are looking to process data closer to where people produce and consume information, rather than relying solely on the enterprise architecture to accommodate increasing volumes of user traffic. This approach delivers multiple benefits:

• Timely insights
• Reliable, increased performance
• Increased productivity
• Enhanced security

DOD must continually improve and optimize data management to reach its goals. To establish an enterprise data mesh and embody global governance, defense leaders must rally together to create this new standard and embrace the concept of a federated data catalog. This may require systems and organizations to absorb capabilities from redundant domains to streamline the shift to data as a product.

We’re seeing the success of implementing the data mesh concept on a microdomain scale and see its potential to speed and pioneer larger transformation across DOD domains. As the data mesh grows, the joint enterprise data fabric can create an agnostic connection framework to reduce the copies of data across the enterprise, rapidly making data-centricity a reality.