Automated Data Quality Assessment Using Machine Learning

Abstract

Data quality is a critical factor in modern data-driven systems, influencing decision-making, predictive modeling, and operational efficiency across domains such as healthcare, finance, and smart cities. Traditional data quality assessment methods, which rely on rule-based frameworks and manual reviews, are often time-consuming, error-prone, and unable to handle large and dynamic datasets. This paper proposes an automated data quality measurement framework using machine learning techniques, focusing on scalability, adaptability, and accuracy. The proposed system integrates supervised and unsupervised learning models to detect anomalies, inconsistencies, missing values, and semantic errors in structured data. It evaluates key data quality dimensions, including completeness, consistency, accuracy, timeliness, and validity, using algorithms such as Decision Trees, Random Forests, Support Vector Machines, and clustering techniques like K-Means and DBSCAN. Feature engineering is applied to capture complex data patterns, while a hybrid approach combining statistical profiling and predictive modeling enhances detection capabilities. The framework includes data preprocessing, feature extraction, model training, evaluation, and deployment stages. Experimental results using benchmark datasets demonstrate improved performance based on precision, recall, F1-score, and accuracy, outperforming traditional rule-based methods. The study highlights challenges such as data heterogeneity, scalability, and model interpretability, and suggests future improvements. Overall, the proposed approach offers a scalable, domain-independent solution that enhances data reliability and reduces manual effort, contributing to efficient data quality management.