Serverless Data Engineering: Leveraging Cloud Technologies for Cost-Effective Data Workflows

  • Authors

    • Zainab Abdullahi Department of Education, Kano State University, Nigeria Author

    DOI:

    https://doi.org/10.67228/30713315/IJAIDT-2022PI1M8G

    Published 04-05-2022

  • Serverless Computing, Cloud Data Engineering, Cost-Effective Data Workflows, Aws Lambda, Google Cloud Functions, Event-Driven Processing, Serverless Data Pipelines, Automated Scaling, Cloud Computing, Data Transformation

    Issue

    Vol. 5 No. 1 (2022)

    Section

    Articles

    How to Cite

    [1]
    Z. Abdullahi, “Serverless Data Engineering: Leveraging Cloud Technologies for Cost-Effective Data Workflows”, IJAIDT, vol. 5, no. 1, pp. 01–10, Apr. 2022, doi: 10.67228/30713315/IJAIDT-2022PI1M8G.

  • Abstract

    Serverless data engineering has emerged as a transformative approach to handling data workflows, offering scalable, cost-efficient, and automated data processing in cloud environments. This paradigm eliminates infrastructure management, enabling data engineers to focus on analytics, data transformations, and insights rather than server maintenance. The shift from traditional data processing to serverless architectures is driven by cost savings, operational efficiency, and scalability. This paper explores serverless data engineering principles, architectures, benefits, and challenges, with a particular focus on real-world applications across industries. We analyze existing cloud-based serverless solutions, including AWS Lambda, Google Cloud Functions, and Azure Functions, for data ingestion, transformation, and analytics. A comparative study of serverless vs. traditional data pipelines is provided, highlighting cost reductions, performance improvements, and automation benefits. Furthermore, we propose a framework for optimizing serverless data workflows, integrating event-driven processing, automated scaling, and security considerations. The paper concludes with a discussion on the future of serverless data engineering and recommendations for effective implementation in organizations seeking cost-effective and efficient data strategies.

  • References

    [1] Jonas, E., Schleier-Smith, J., Sreekanti, V., et al. (2019). Cloud Programming Simplified: A Berkeley View on Serverless Computing. arXiv.

    [2] Müller, I., Marroquín, R., Alonso, G. (2019). Lambada: Interactive Data Analytics on Cold Data using Serverless Cloud Infrastructure. arXiv.

    [3] García-López, P., Arjona, A., Sampe, J., et al. (2020). Triggerflow: Trigger-based Orchestration of Serverless Workflows. arXiv.

    [4] Eismann, S., Scheuner, J., van Eyk, E., et al. (2020). A Review of Serverless Use Cases and their Characteristics. arXiv.

    [5] Rahman, M. M., Hasan, M. H. (2019). Serverless Architecture for Big Data Analytics. IEEE GCAT Conference.

    [6] Pogiatzis, A., Samakovitis, G. (2020). An Event-Driven Serverless ETL Pipeline on AWS. Applied Sciences.

    [7] Toader, L., Uta, A., Musaafir, A., Iosup, A. (2019). Graphless: Toward Serverless Graph Processing. IEEE ISPDC.

    [8] Carver, B., Zhang, J., Wang, A., et al. (2020). LADS: A High-Performance Framework for Serverless Parallel Computing. ACM SoCC.

    [9] Aytekin, A., Johansson, M. (2019). Harnessing the Power of Serverless Runtimes for Large-Scale Optimization. arXiv.

    [10] Kumanov, D., Hung, L. H., Lloyd, W., Yeung, K. Y. (2018). Serverless Computing for High-Performance Biomedical Research. arXiv.

    [11] Hung, L. H., Kumanov, D., Niu, X., et al. (2019). Rapid RNA Sequencing Data Analysis using Serverless Computing. bioRxiv.

    [12] Lee, B. D., Timony, M. A., Ruiz, P. (2019). DNAVisualization.org: A Serverless Web Tool for DNA Sequence Visualization. Nucleic Acids Research.

    [13] Kratzke, N., Peinl, R. (2017). ClouNS: A Cloud-Native Application Reference Model for Enterprise Architects. arXiv.

    [14] Baldini, I., Castro, P., Chang, K., et al. (2017). Serverless Computing: Current Trends and Open Problems. IBM Research.

    [15] Spillner, J. (2017). Serverless Computing and Applications: Current Trends and Open Problems. IEEE Cloud Computing.

  • Downloads