Design and Evaluation of an Automated Data Quality and Governance Framework for Large-Scale Enterprise Data

Abstract

The growing volume, diversity, and velocity of enterprise data create challenges in maintaining data quality, consistency, traceability, and governance. Conventional approaches often depend on fragmented validation rules, manual inspection, and isolated governance mechanisms, limiting continuous quality management at scale. This study proposes an automated data quality and governance framework integrating continuous data profiling, quality validation, anomaly detection, automated remediation, governance-policy enforcement, metadata management, and data lineage within a unified data engineering architecture. The framework identifies missing values, duplicates, invalid records, schema violations, inconsistencies, and anomalous observations across enterprise data sources. An experimental evaluation uses controlled datasets containing systematically introduced quality defects. Performance is assessed using precision, recall, F1-score, data-quality improvement, remediation success rate, governance compliance, lineage completeness, and throughput, latency, and resource utilization. Simulated results indicate effective defect detection and remediation, strong governance and lineage coverage, and scalable processing performance. The framework provides a reproducible foundation for trustworthy, scalable, and maintainable enterprise data platforms.