Intelligent Data Integration and Workload Optimization for Scalable Distributed Analytics Systems

  • Authors

    • Faruck Mohammed Research Analyst, India. Author

    DOI:

    https://doi.org/10.67228/30716357/IJMRSE-V9I3P104

    Published 08-24-2026

  • Intelligent Data Integration, Distributed Analytics, Workload Optimization, Semantic Interoperability, Adaptive Resource Allocation, Data Quality, Distributed Query Processing, Cloud Computing

    Issue

    Vol. 9 No. 3 (2026)

    Section

    Articles

  • Abstract

    The main purpose of this paper is to develop a theoretical framework that will allow organizations to improve their distributed enterprise analytics capabilities by integrating semantic data; managing metadata; controlling for data quality; optimizing queries; and adaptively allocating resources. The objective of the framework is to increase both the scalability and resource efficiency of the distributed data environment as well as the reliability of the analytics performed on the data. To evaluate how effectively the framework can achieve these goals, we assess its performance theoretically in terms of latency, throughput, resource utilization, data quality, fault recovery, and operating costs. While there were no experimental tests performed in this study, the advantages claimed by the framework must be validated through empirical testing.

  • References

    [1] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113. https://doi.org/10.1145/1327452.1327492

    [2] Yachamaneni, T., Kotadiya, U., & Arora, A. S. (2021). Enhancing Data Throughput and Latency in Distributed In-Memory Systems for AI-Driven Applications across Public Cloud Infrastructure. International Journal of AI, BigData, Computational and Management Studies, 2(4), 69-79.

    [3] Gantikota, S. (2023). Integrating SonarQube and IBM AppScan into Enterprise CI/CD Pipelines: A Vulnerability Mitigation Framework Achieving Over Eighty Percent Risk Reduction. International Journal of Emerging Trends in Computer Science and Information Technology, 4(3), 240-244. https://doi.org/10.63282/3050-9246.IJETCSIT-V4I3P124

    [4] Doan, A., Madhavan, J., Domingos, P., & Halevy, A. (2005). Semantic integration research in the database community: A brief survey. AI Magazine, 26(1), 83–94. https://doi.org/10.1609/aimag.v26i1.1801

    [5] Halevy, A. Y. (2001). Answering queries using views: A survey. The VLDB Journal, 10, 270–294. https://doi.org/10.1007/s007780100054

    [6] Rahm, E., & Bernstein, P. A. (2001). A survey of approaches to automatic schema matching. The VLDB Journal, 10, 334–350. https://doi.org/10.1007/s007780100057

    [7] Paruchuri, J. K. (2021). Lakehouse architecture: Unifying data lakes and data warehouses. International Journal of Computer Techniques (IJCT), 8(1), 1-15.

    [8] Rocklin, M. (2015). Dask: Parallel computation with blocked algorithms and task scheduling. In Proceedings of the 14th Python in Science Conference (pp. 130–136). https://doi.org/10.25080/Majora-7b98e3ed-013

    [9] Semantic data integration and querying: A survey and challenges. (2024). ACM Computing Surveys, 56(8), Article 209. https://doi.org/10.1145/3653317

    [10] Sandra, K. (2024). THE REGULATED BANKING AI LAKEHOUSE. INDO-CONTINENTAL ACADEMIC PUBLISHERS.

    [11] Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., & Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12) (pp. 15–28). USENIX Association. https://www.usenix.org/conference/nsdi12/technical-sessions/presentation/zaharia

    [12] Sunkara, R. (2024). Improving Observability and Stability in Wayland-Based Compositors: Lifecycle Logging, Buffer Validation, and Crash Hardening in Production Display Stacks. American International Journal of Computer Science and Technology, 6(1), 60-64. https://doi.org/10.63282/3117-5481/AIJCST-V6I1P106

    [13] Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., Ghodsi, A., Gonzalez, J., Shenker, S., & Stoica, I. (2016). Apache Spark: A unified engine for big data processing. Communications of the ACM, 59(11), 56–65. https://doi.org/10.1145/2934664

    [14] Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., & Whittle, S. (2015). The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment, 8(12), 1792–1803. https://doi.org/10.14778/2824032.2824076

    [15] Arora, A. S., Yachamaneni, T., & Kotadiya, U. (2024). Architectural Optimization of Serverless Big Data Pipelines for AI Workloads Using Cloud Functions and Managed Spark on GCP. International Journal of Emerging Trends in Computer Science and Information Technology, 5(1), 61-68.

    [16] Chaudhuri, S. (1998). An overview of query optimization in relational systems. In Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems (pp. 34–43). Association for Computing Machinery. https://doi.org/10.1145/275487.275492

    [17] Brahmandam, L. M. K. (2025). Design Patterns and Empirical Evaluation of Reusable Terraform Modules Encoding Audit-Ready Defaults for Multi-Account AWS Deployments: A Cross-Team Study across EC2, S3, RDS, EKS, IAM, and Cloud Watch. International Journal of Emerging Research in Engineering and Technology, 6(2), 133-142. https://doi.org/10.63282/3050-922X.IJERET-V6I2P116

  • Downloads