Integrating Lakehouse Architecture with Data Sovereignty-Compliant Big Data Workloads

  • Authors

    • N. Seshagiri Information Technology Pioneer, National Informatics Centre, India. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2023PI7V1F

    Published 05-09-2023

  • Lakehouse Architecture, Data Sovereignty, Big Data, Data Governance, Compliance, Distributed Data Systems, Multi-Cloud, Data Pipelines, Data Security, Data Warehousing, Regulatory Compliance, Cloud Data Platforms

    Issue

    Vol. 6 No. 1 (2023)

    Section

    Articles

    How to Cite

    [1]
    S. N, “Integrating Lakehouse Architecture with Data Sovereignty-Compliant Big Data Workloads”, IJDEIC, vol. 6, no. 1, pp. 01–10, May 2023, doi: 10.67228/30715717/IJDEIC-2023PI7V1F.

  • Abstract

    The increasing complexity of data governance and regulatory compliance in an era of distributed, global data systems has made data sovereignty a critical consideration for modern data architectures. This paper explores the integration of Lakehouse Architecture with data sovereignty-compliant frameworks to efficiently manage big data workloads across multiple jurisdictions. By combining the flexibility and scalability of data lakes with the management and query optimization capabilities of data warehouses, Lakehouse architecture offers a unified solution that ensures both agility and compliance. We delve into the challenges and strategies of enforcing data sovereignty in multi-region and multi-cloud environments while maintaining high performance and security in big data operations. Additionally, we examine best practices, tools, and frameworks that support data sovereignty in Lakehouse implementations, offering a comprehensive guide to building compliant data pipelines and analytics systems.

  • References

    [1] European Commission. (2018). General Data Protection Regulation (GDPR). [Online] Available at: https://ec.europa.eu/info/law/law-topic/data-protection_en

    [2] California Legislative Information. (2018). California Consumer Privacy Act (CCPA). [Online] Available at: https://www.oag.ca.gov/privacy/ccpa

    [3] Shaw, M. J., & Appelbaum, D. (2020). "Big Data Privacy and Security Issues in the Age of Cloud Computing". Journal of Data Privacy and Security, 16(2), 34-52.

    [4] Hadoop, A., & Spark, M. (2019). "Data Governance and Compliance: Challenges and Strategies". Journal of Big Data Management, 6(3), 21-34.

    [5] Smith, R. E., & Wilson, K. (2019). "Managing Data Sovereignty in Cloud Environments". Cloud Computing Journal, 12(1), 56-67.

    [6] Amazon Web Services. (2020). AWS Data Encryption and Compliance Features. [Online] Available at: https://aws.amazon.com/compliance/

    [7] Microsoft Azure. (2020). Azure Security and Compliance. [Online] Available at: https://azure.microsoft.com/en-us/overview/security/

    [8] Velte, A. T., & Velte, T. J. (2020). Cloud Computing: A Practical Approach. McGraw-Hill.

    [9] Kwon, J., & Kim, H. (2021). "Data Sovereignty and the Impact on Big Data Architectures". Data Science and Engineering, 7(4), 85-102.

    [10] Gartner, Inc. (2020). Magic Quadrant for Cloud Database Management Systems. [Online] Available at: https://www.gartner.com/en/documents/3980728.

  • Downloads