Metadata-Driven Data Transformation in Data Mesh Architectures

  • Authors

    • Dr. Lei Weing Department of Data Science, Tsinghua University, Beijing, China. Author
    • Dr. Chen Wei Professor, Peking University, China. Author

    DOI:

    https://doi.org/10.67228/30713315/IJAIDT-2021PI7T1E

    Published 02-03-2021

  • Data Mesh, Metadata-Driven Architecture, Data Transformation, Data Governance, Active Metadata, Data Pipelines, Decentralized Data Architecture, Domain-Oriented Data Ownership, Data Quality, Interoperability

    Issue

    Vol. 4 No. 1 (2021)

    Section

    Articles

    How to Cite

    [1]
    L. Weing and C. Wei, “Metadata-Driven Data Transformation in Data Mesh Architectures”, IJAIDT, vol. 4, no. 1, pp. 01–11, Feb. 2021, doi: 10.67228/30713315/IJAIDT-2021PI7T1E.

  • Abstract

    In the evolving landscape of data architectures, the Data Mesh paradigm has emerged as a scalable and decentralized approach to managing complex data ecosystems. However, as data domains grow increasingly autonomous, ensuring consistency, discoverability, and transformation standardization across domains becomes challenging. This paper proposes a metadata-driven approach to data transformation within Data Mesh architectures. By leveraging active metadata such as data lineage, transformation rules, quality metrics, and semantic definitions data domains can automate and orchestrate transformations in a consistent, governed, and scalable manner. The paper explores architectural patterns, implementation considerations, and case studies demonstrating the benefits of this approach in enhancing interoperability, governance, and agility in enterprise data systems.

  • References

    [1] Zhamak Dehghani, Data Mesh Principles and Logical Architecture, (2020).

    [2] Inês Araújo Machado, Carlos Costa, Maribel Yasmina Santos, Data Driven Information Systems: The Data Mesh Paradigm Shift, ISD 2021.

    [3] Pegdwendé Sawadogo et al., Metadata Systems for Data Lakes: Models and Features, MEDAL graph based metadata for lakes (2019).

    [4] Nogueira, I., Romdhane, M., & Darmont, J. (2018). Modeling Data Lake Metadata with a Data Vault. arXiv:1807.04035.

    [5] Diamantini, C., Lo Giudice, P., Musarella, L., Potena, D., Storti, E., & Ursino, D. (2018). A New Metadata Model to Uniformly Handle Heterogeneous Data Lake Sources.

    [6] Munshi, A. A., & Mohamed, Y. A.-R. I. (2018). Data Lake Lambda Architecture for Smart Grid Big Data Analytics. IEEE Access.

    [7] Ulrich, H., Kock-Schoppenhauer, A.-K., Deppenwiese, N., et al. (2018). Understanding the Nature of Metadata: A Systematic Review.

    [8] Aisyah, M., & Ruldeviyani, Y. (2018). Designing Data Governance Structure Based on DMBOK Framework.

    [9] Al-Ruithe, M., Benkhelifa, E., & Hameed, K. (2018). Data Governance Taxonomy: Cloud versus Non-Cloud.

    [10] Beheshti, A., Benatallah, B., Nouri, R., & Chhieng, V. (2018). CoreKG: A Knowledge Lake Service.

    [11] Hai, R., Quix, C., & Jarke, M. (2016/2018 cited). Data Lake Concept and Systems for Big Data Management.

    [12] Khine, P. P., & Wang, Z. S. (2018). Data Lake: A New Ideology in Big Data Era.

    [13] Greenberg, J. (2017/2018 cited). Big Metadata, Smart Metadata, and Metadata Capital.

    [14] Gorelik, A. (2019 pre-work, earlier concepts used). The Enterprise Big Data Lake (conceptual foundations prior to 2019).

    [15] Dutta, H. (2016/2018 relevance). Graph-Based Data Governance Model for Real-Time Data Ingestion.

  • Downloads