Serverless Data Engineering: Leveraging Cloud Technologies for Cost-Effective Data Workflows

  • Authors

    • Dr. Peter Naur Professor, University of Copenhagen, Denmark. Author
    • John Backus Research Scientist, IBM Research, United States. Author
    • Donald Davies Scientist, National Physical Laboratory, United Kingdom. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2024PI9X6C

    Published 06-10-2024

  • Serverless Computing, Cloud Data Engineering, Cost-Effective Data Workflows, AWS Lambda, Google Cloud Functions, Event-Driven Processing, Serverless Data Pipelines, Automated Scaling, Cloud Computing, Data Transformation

    Issue

    Vol. 7 No. 1 (2024)

    Section

    Articles

    How to Cite

    [1]
    P. Naur, J. Backus, and D. Davies, “Serverless Data Engineering: Leveraging Cloud Technologies for Cost-Effective Data Workflows”, IJDEIC, vol. 7, no. 1, pp. 01–09, Jun. 2024, doi: 10.67228/30715717/IJDEIC-2024PI9X6C.

  • Abstract

    Serverless data engineering has emerged as a transformative approach to handling data workflows, offering scalable, cost-efficient, and automated data processing in cloud environments. This paradigm eliminates infrastructure management, enabling data engineers to focus on analytics, data transformations, and insights rather than server maintenance. The shift from traditional data processing to serverless architectures is driven by cost savings, operational efficiency, and scalability. This paper explores serverless data engineering principles, architectures, benefits, and challenges, with a particular focus on real-world applications across industries. We analyze existing cloud-based serverless solutions, including AWS Lambda, Google Cloud Functions, and Azure Functions, for data ingestion, transformation, and analytics. A comparative study of serverless vs. traditional data pipelines is provided, highlighting cost reductions, performance improvements, and automation benefits. Furthermore, we propose a framework for optimizing serverless data workflows, integrating event-driven processing, automated scaling, and security considerations. The paper concludes with a discussion on the future of serverless data engineering and recommendations for effective implementation in organizations seeking cost-effective and efficient data strategies.

  • References

    [1] Hassan, H. B., Barakat, S. A., & Sarhan, Q. I. (2021). Survey on serverless computing. Journal of Cloud Computing, 10(39).

    [2] Baldini, I., Castro, P., Chang, K., et al. (2017). Serverless Computing: Current Trends and Open Problems. Research Advances in Cloud Computing.

    [3] Jonas, E., Schleier-Smith, J., Sreekanti, V., et al. (2019). Cloud Programming Simplified: A Berkeley View on Serverless Computing. arXiv.

    [4] Lloyd, W., Ramesh, S., Chinthalapati, S., Ly, L., & Pallickara, S. (2018). Serverless Computing: An Investigation of Factors Influencing Microservice Performance. IEEE Cloud Engineering.

    [5] McGrath, G., & Brenner, P. (2017). Serverless Computing: Design, Implementation, and Performance. IEEE International Conference on Distributed Computing Systems Workshops.

    [6] Spillner, J. (2017). Snafu: Function-as-a-Service (FaaS) Runtime Design and Implementation.

    [7] Akkus, I., Chen, R., et al. (2018). SAND: Towards High-Performance Serverless Computing. USENIX ATC.

    [8] Shahrad, M., Fonseca, R., et al. (2020). Serverless in the Wild: Characterizing and Optimizing the Serverless Workload. USENIX ATC.

    [9] García-López, P., et al. (2020). Triggerflow: Trigger-Based Orchestration of Serverless Workflows. arXiv.

    [10] Gajula, S. (2023). A review of anomaly identification in finance frauds using machine learning system. International Journal of Current Engineering and Technology, 13(6), 568–575. https://ijcet.evegenis.org/index.php/ijcet/article/view/820

    [11] Burckhardt, S., Gillum, C., et al. (2021). Serverless Workflows with Durable Functions and Netherite. arXiv.

    [12] Wen, J., & Liu, Y. (2021). An Empirical Study on Serverless Workflow Services. arXiv.

    [13] Rahman, M. M., & Hasan, M. H. (2019). Serverless Architecture for Big Data Analytics. IEEE Conference Proceedings.

    [14] Taibi, D., Lenarduzzi, V., & Pahl, C. (2020). Patterns for Serverless Functions (Function-as-a-Service): A Multivocal Literature Review.

    [15] Shahane, V. (2022). Serverless Computing in Cloud Environments: Architectural Patterns and Performance Optimization.

    [16] Bansal, I. (2023). Event-Driven Machine Learning Infrastructure Using Serverless Cloud Functions.

  • Downloads