Multi-Agent Systems for Autonomous Data Pipeline Optimization in AI Workflows

  • Authors

    • José María Troya Computer Architecture, University of Málaga, Spain. Author
    • Ramon López de Mántaras AI Research Pioneer, IIIA-CSIC, Spain. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2023PII1N7X

    Published 08-02-2023

  • Multi-Agent Systems, Data Pipeline Optimization, Autonomous Systems, AI Workflows, Distributed Data Processing, Workflow Management, Adaptive Scheduling, Resource Optimization

    Issue

    Vol. 6 No. 2 (2023)

    Section

    Articles

    How to Cite

    [1]
    J. M. Troya and R. L. de Mántaras, “Multi-Agent Systems for Autonomous Data Pipeline Optimization in AI Workflows”, IJDEIC, vol. 6, no. 2, pp. 01–10, Aug. 2023, doi: 10.67228/30715717/IJDEIC-2023PII1N7X.

  • Abstract

    The optimization of data pipelines is critical for enhancing the performance and efficiency of AI workflows, which often involve complex, heterogeneous, and dynamic data processing stages. Traditional approaches to pipeline optimization struggle to adapt autonomously to evolving workloads and system conditions. This paper proposes a novel multi-agent system (MAS) framework that enables autonomous optimization of data pipelines in AI workflows. Each agent is responsible for specific tasks such as data ingestion, transformation, scheduling, and resource management, and they collaborate through adaptive protocols to achieve global optimization objectives. We demonstrate the effectiveness of the proposed framework through extensive experiments, showing significant improvements in pipeline throughput, latency, and resource utilization compared to conventional methods. Our approach highlights the potential of MAS to bring intelligence, flexibility, and scalability to data pipeline management in AI systems.

  • References

    [1] M. Wooldridge; An Introduction to MultiAgent Systems; Wiley; 2009.

    [2] J. Ferber; Multi-Agent Systems: An Introduction to Distributed Artificial Intelligence; Addison-Wesley; 1999.

    [3] R. Buyya; D. Abramson; and J. Giddy; "An economy driven resource management architecture for global computational power sharing;" Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications; 2000.

    [4] S. Russell and P. Norvig; Artificial Intelligence: A Modern Approach; 4th Edition; Pearson; 2020.

    [5] M. Stonebraker et al.; "The architecture of SciDB;" CIDR; 2011.

    [6] T. White; Hadoop: The Definitive Guide; O’Reilly Media; 2015.

    [7] F. Bonomi et al.; "Fog Computing and Its Role in the Internet of Things;" Proceedings of the MCC Workshop on Mobile Cloud Computing; 2012.

    [8] A. Varga et al.; "On the Design of Adaptive Data Pipelines for Real-Time Analytics;" IEEE Transactions on Cloud Computing; 2018.

    [9] L. Pan et al.; "Data Pipeline Optimization Using Reinforcement Learning;" Proceedings of the ACM SIGMOD International Conference on Management of Data; 2020.

    [10] K. Decker and V. Lesser; "Designing a Family of Coordination Algorithms;" Proceedings of the First International Conference on Multi-Agent Systems; 1995.

    [11] G. Weiss (Ed.); Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence; MIT Press; 1999.

    [12] S. Girdzijauskas and J. M. Kristensen; "An agent-based framework for data processing in distributed systems;" IEEE Transactions on Systems; Man; and Cybernetics; 2012.

    [13] D. K. Hsu et al.; "Adaptive Resource Scheduling in Cloud Computing Using Multi-Agent Reinforcement Learning;" IEEE Transactions on Services Computing; 2021.

    [14] H. Casanova et al.; "Workflow Scheduling and Resource Management in Distributed Computing Environments;" Proceedings of the IEEE; 2014.

    [15] J. Dean and S. Ghemawat; "MapReduce: Simplified Data Processing on Large Clusters;" Communications of the ACM; 2008.

  • Downloads