AI-Based Resource Scheduling in Distributed Data Engineering Platforms

  • Authors

    • Louis Pouzin Computer Network Researcher, IRIA, France. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2018PI8F1Q

    Published 10-04-2018

  • Artificial Intelligence, Resource Scheduling, Distributed Computing, Data Engineering, Machine Learning, Cloud Computing, Intelligent Scheduling, Distributed Analytics, Big Data Processing, Workload Optimization, Resource Allocation, Predictive Analytics

    Issue

    Vol. 1 No. 2 (2018)

    Section

    Articles

    How to Cite

    [1]
    L. Pouzin, “AI-Based Resource Scheduling in Distributed Data Engineering Platforms”, IJDEIC, vol. 1, no. 2, pp. 01–22, Oct. 2018, doi: 10.67228/30715717/IJDEIC-2018PI8F1Q.

  • Abstract

    Modern distributed data engineering platforms process massive and diverse datasets, making efficient resource scheduling increasingly challenging. Traditional scheduling algorithms struggle to adapt to dynamic workloads and heterogeneous computing environments, resulting in poor resource utilization and increased execution time. This paper proposes an AI-Based Resource Scheduling Framework that integrates workload prediction, intelligent resource allocation, adaptive scheduling, and continuous performance monitoring. By leveraging machine learning and reinforcement learning, the framework dynamically optimizes scheduling decisions based on workload characteristics, resource availability, and real-time system feedback. Experimental results demonstrate improved resource utilization, reduced scheduling latency, enhanced scalability, lower operational costs, and better workload balancing compared to conventional scheduling approaches, making the framework well-suited for cloud-native and large-scale distributed data engineering environments.

  • References

    [1] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, "A View of Cloud Computing," Communications of the ACM, vol. 53, no. 4, pp. 50–58, Apr. 2010.

    [2] I. Foster, Y. Zhao, I. Raicu, and S. Lu, "Cloud Computing and Grid Computing 360-Degree Compared," Proceedings of the Grid Computing Environments Workshop, pp. 1–10, 2008.

    [3] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica, "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing," Proc. NSDI, pp. 15–28, 2012.

    [4] B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. Joseph, R. Katz, S. Shenker, and I. Stoica, "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center," Proc. NSDI, pp. 295–308, 2011.

    [5] V. Mao, M. Alizadeh, and I. Menache, "Resource Management with Deep Reinforcement Learning," Proceedings of ACM HotNets, pp. 50–56, 2016.

    [6] R. Sutton and A. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA, USA: MIT Press, 2018.

    [7] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to Algorithms, 3rd ed. Cambridge, MA, USA: MIT Press, 2009.

    [8] J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Communications of the ACM, vol. 51, no. 1, pp. 107–113, Jan. 2008.

    [9] M. Satyanarayanan, "The Emergence of Edge Computing," Computer, vol. 50, no. 1, pp. 30–39, Jan. 2017.

    [10] H. Topcuoglu, S. Hariri, and M. Wu, "Performance-Effective and Low-Complexity Task Scheduling for Heterogeneous Computing," IEEE Transactions on Parallel and Distributed Systems, vol. 13, no. 3, pp. 260–274, Mar. 2002.

    [11] A. Verma, G. Das, T. K. Nayak, P. De, and R. Kothari, "Server Workload Analysis for Power Minimization Using Consolidation," USENIX Annual Technical Conference, pp. 28–41, 2009.

    [12] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, "A Survey on Mobile Edge Computing: The Communication Perspective," IEEE Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2322–2358, 2017.

    [13] A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica, "Dominant Resource Fairness: Fair Allocation of Multiple Resource Types," Proceedings of NSDI, pp. 323–336, 2011.

  • Downloads