Intelligent Data Provenance Tracking for Large-Scale Analytics Platforms

Abstract

Large-scale analytics platforms generate massive amounts of data from diverse sources, making data transparency, accountability, and governance increasingly important. Data provenance, which records the origin, transformation, ownership, and movement of data, plays a critical role in ensuring data quality, reproducibility, compliance, and trustworthiness. However, traditional provenance tracking methods often struggle to handle the complexity and scale of modern analytics environments. This study proposes an Intelligent Data Provenance Tracking Framework (IDPTF) that integrates artificial intelligence, graph-based metadata management, distributed provenance storage, and automated lineage analysis. The framework consists of four layers: Data Acquisition, Provenance Intelligence, Metadata Storage, and Analytics Governance, enabling end-to-end data traceability with minimal performance overhead. Using graph databases and machine learning techniques, the framework captures provenance information across the entire data lifecycle, detects anomalies, optimizes metadata storage, and reconstructs data lineage dynamically. Experimental results demonstrate improved provenance completeness, faster query performance, enhanced governance effectiveness, simplified compliance auditing, and greater resilience to data inconsistencies. The AI-driven approach also enables proactive risk prediction, supporting trusted analytics and reliable decision-making. Overall, the proposed framework provides a scalable and flexible solution for provenance management in modern analytics ecosystems, with future enhancements including blockchain integration, federated learning support, and autonomous governance mechanisms.