Architecting Real-time Predictive Analytics Pipelines Using Snowflake and AWS Lambda for IoT Data Streams

  • Authors

    • Ritu Agarwal Finance Manager, Capgemini, India. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2018PII7Y4D

    Published 11-08-2018

  • Real-Time Analytics, Iot Data Streams, AWS Lambda, Snowflake, Serverless Architecture, Predictive Analytics, Edge Computing, Data Pipeline, Streaming Data, Cloud Computing

    Issue

    Vol. 1 No. 2 (2018)

    Section

    Articles

  • Abstract

    As the Internet of Things (IoT) proliferates, vast volumes of streaming sensor data are being generated in real-time, creating both opportunities and challenges for data-driven decision-making. This paper proposes a cloud-native architecture that integrates Snowflake’s scalable data platform with AWS Lambda's serverless compute capabilities to build real-time predictive analytics pipelines for IoT data. By leveraging AWS services for ingestion (such as Kinesis or MQTT over IoT Core), Lambda for event-driven processing, and Snowflake for scalable storage and analysis, the proposed solution enables rapid deployment of machine learning models to process streaming data. The architecture is designed to be low-latency, cost-effective, and easily extensible for a variety of industrial applications. A prototype implementation and performance evaluation are presented, demonstrating the effectiveness of the architecture in handling high-throughput, low-latency IoT workloads.

  • References

    [1] Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2013). Internet of Things (IoT): A vision, architectural elements, and future directions. Future Generation Computer Systems, 29(7), 1645-1660.

    [2] Zaharia, M., Das, T., Li, H., et al. (2013). Discretized streams: Fault-tolerant streaming computation at scale. Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles.

    [3] Li, S., Da Xu, L., & Zhao, S. (2018). The internet of things: a survey. Information Systems Frontiers, 17(2), 243–259.

    [4] AWS. (2024). AWS IoT Core Documentation. https://docs.aws.amazon.com/iot/

    [5] Affetti, L., Tommasini, R., Margara, A., Cugola, G., & Della Valle, E. (2017). Defining the execution semantics of stream processing engines. Journal of Big Data, 4(12), 1–33. https://doi.org/10.1186/s40537-017-0072-9

    [6] Dean, J., & Ghemawat, S. (2008). MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113.

    [7] Fernandes, D. A. B., Soares, L. F. B., Gomes, J. V., et al. (2014). Security issues in cloud environments: a survey. International Journal of Information Security, 13(2), 113–170.

    [8] Zhang, Y., & Wen, J. (2017). An IoT electric business model based on the protocol of Bitcoin. Future Internet, 9(1), 12.

    [9] Kitchin, R. (2014). The real-time city? Big data and smart urbanism. GeoJournal, 79, 1–14.

  • Downloads