Machine Learning-Driven IT Cost Prediction Models in Scalable Data Platforms

  • Authors

    • Dr. Zdzisław Pawlak Professor, Warsaw University of Technology, Poland. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2023PI9D6L

    Published 06-08-2023

  • IT Cost Prediction, Machine Learning, Scalable Data Platforms, Cloud Computing, Time Series Forecasting, Big Data Analytics, Cost Optimization, Resource Management, Predictive Modeling, Distributed Systems

    Issue

    Vol. 6 No. 1 (2023)

    Section

    Articles

    How to Cite

    [1]
    Z. Pawlak, “Machine Learning-Driven IT Cost Prediction Models in Scalable Data Platforms”, IJDEIC, vol. 6, no. 1, pp. 01–12, Jun. 2023, doi: 10.67228/30715717/IJDEIC-2023PI9D6L.

  • Abstract

    In modern IT ecosystems, accurately predicting operational costs is critical for budgeting, resource allocation, and service optimization. Traditional cost estimation models often fall short in environments characterized by high variability and scale. This paper presents a machine learning-driven approach to IT cost prediction, leveraging scalable data platforms to process large volumes of operational data in real time. We explore various regression and time series forecasting models trained on usage metrics and workload patterns, and demonstrate how distributed architectures (e.g., cloud-native platforms, big data processing engines) enhance both the accuracy and scalability of the prediction process. Experimental results show that the proposed models significantly outperform traditional baselines in terms of accuracy and adaptability, offering a viable pathway for proactive cost management in complex IT environments.

  • References

    [1] J. Smith and A. Kumar; "Predictive Cost Optimization in Cloud Services Using Machine Learning;" IEEE Transactions on Cloud Computing; vol. 8; no. 2; pp. 395–408; Apr.–Jun. 2020.

    [2] M. Zaharia et al.; "Apache Spark: A Unified Engine for Big Data Processing;" Communications of the ACM; vol. 59; no. 11; pp. 56–65; Nov. 2016.

    [3] A. Paszke et al.; "PyTorch: An Imperative Style; High-Performance Deep Learning Library;" Advances in Neural Information Processing Systems; vol. 32; 2019.

    [4] D. Sculley et al.; "Hidden Technical Debt in Machine Learning Systems;" in Proc. NIPS; 2015.

    [5] T. Chen and C. Guestrin; "XGBoost: A Scalable Tree Boosting System;" in Proc. 22nd ACM SIGKDD; 2016; pp. 785–794.

    [6] H. Seaborn et al.; "Forecasting Cloud Infrastructure Costs Using LSTM Networks;" ACM Transactions on Management Information Systems; vol. 11; no. 4; 2021.

    [7] F. Chollet; "Deep Learning with Python;" Manning Publications; 2017.

    [8] A. Taly et al.; "Explainability in AI: Interpretations for Tree-Based and Deep Models;" arXiv preprint arXiv:2005.01789; 2020.

    [9] T. Wolf et al.; "Transformers: State-of-the-Art Natural Language Processing;" in Proc. EMNLP 2020: Demos; pp. 38–45.

    [10] L. Breiman; "Random Forests;" Machine Learning; vol. 45; pp. 5–32; 2001.

    [11] M. Abadi et al.; "TensorFlow: A System for Large-Scale Machine Learning;" in Proc. OSDI; 2016.

    [12] R. Tibshirani; "Regression Shrinkage and Selection via the Lasso;" Journal of the Royal Statistical Society; Series B; vol. 58; no. 1; pp. 267–288; 1996.

    [13] Facebook; "Prophet: Forecasting at Scale;" [Online]. Available: https://facebook.github.io/prophet

    [14] R. B. Cleveland et al.; "STL: A Seasonal-Trend Decomposition Procedure Based on Loess;" Journal of Official Statistics; vol. 6; no. 1; pp. 3–73; 1990.

    [15] S. Ruder; "An Overview of Gradient Descent Optimization Algorithms;" arXiv preprint arXiv: 1609.04747; 2016.

  • Downloads