Intelligent Workflow Scheduling for Distributed Data Processing Systems

  • Authors

    • Dr. David Parnas Professor, McMaster University, Canada. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2024PII8H5A

    Published 12-04-2024

  • Distributed Systems, Workflow Scheduling, Intelligent Scheduling, Resource Allocation, Big Data Processing, Load Balancing, Optimization Algorithms, Cloud Computing

    Issue

    Vol. 7 No. 2 (2024)

    Section

    Articles

    How to Cite

    [1]
    D. Parnas, “Intelligent Workflow Scheduling for Distributed Data Processing Systems”, IJDEIC, vol. 7, no. 2, pp. 01–13, Dec. 2024, doi: 10.67228/30715717/IJDEIC-2024PII8H5A.

  • Abstract

    The rapid growth of data-intensive applications in scientific computing, enterprise analytics, and cloud services has increased the demand for efficient distributed data processing systems. Traditional scheduling methods like FCFS, Round Robin, and heuristic approaches often fail to meet the dynamic and heterogeneous requirements of modern environments. This paper proposes an intelligent workflow scheduling framework that improves performance through adaptive decision-making, predictive analytics, and machine learning. The system dynamically allocates tasks based on resource availability, workflow dependencies, and historical execution data, enabling it to anticipate bottlenecks and reassign tasks proactively. It also incorporates resource heterogeneity modeling and dependency-aware scheduling to reduce idle time and optimize execution. Performance is evaluated using metrics such as makespan, throughput, resource utilization, and fault tolerance, showing significant improvements over traditional methods. The framework also addresses key challenges like load balancing, scalability, energy efficiency, and fault tolerance. Overall, the proposed approach enhances system efficiency and scalability while supporting integration with emerging technologies such as edge computing and hybrid cloud environments, paving the way for more autonomous and resilient distributed scheduling systems.

  • References

    [1] Abraham Silberschatz, Peter B. Galvin, & Greg Gagne (2018). Operating System Concepts (10th ed.). Wiley.

    [2] Michael J. Quinn (2004). Parallel Programming in C with MPI and OpenMP. McGraw-Hill.

    [3] Maheswaran M., et al. (1999). Dynamic mapping of a class of independent tasks onto heterogeneous computing systems. Journal of Parallel and Distributed Computing.

    [4] Tarek El-Ghazawi, et al. (2005). UPC: Distributed Shared Memory Programming. Wiley.

    [5] H. Topcuoglu, Salim Hariri, & Min-You Wu (2002). Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Transactions on Parallel and Distributed Systems.

    [6] Rajkumar Buyya, Chee Shin Yeo, & Srikumar Venugopal (2009). Market-oriented cloud computing: Vision and directions. Future Generation Computer Systems.

    [7] Gajula, S. (2023). A review of anomaly identification in finance frauds using machine learning system. International Journal of Current Engineering and Technology, 13(6), 568–575. https://ijcet.evegenis.org/index.php/ijcet/article/view/820

    [8] David E. Goldberg (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley.

    [9] James Kennedy & Russell Eberhart (1995). Particle swarm optimization. Proceedings of IEEE International Conference on Neural Networks.

    [10] Marco Dorigo (1996). Ant colony optimization for combinatorial problems. IEEE Transactions on Systems, Man, and Cybernetics.

    [11] E. Alba & B. Dorronsoro (2008). Cellular Genetic Algorithms. Springer.

    [12] Ian Foster, Carl Kesselman (2003). The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann.

    [13] Thomas L. Casavant & Jon G. Kuhl (1988). A taxonomy of scheduling in general-purpose distributed computing systems. IEEE Transactions on Software Engineering.

    [14] Kevin P. Murphy (2012). Machine Learning: A Probabilistic Perspective. MIT Press.

    [15] Richard S. Sutton & Andrew G. Barto (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.

    [16] Qiang Wu, et al. (2013). A survey of scheduling in cloud computing systems. Journal of Network and Computer Applications.

  • Downloads