Continuous Data Transformation in Event-Driven Microservices

  • Authors

    • Prof. Thabo Nkosi Faculty of Engineering, Johannesburg Technical University, South Africa. Author

    DOI:

    https://doi.org/10.67228/30713315/IJAIDT-2019PII0Q75

    Published 08-05-2019

  • Event-Driven Architecture, Microservices, Continuous Data Transformation, Stream Processing, Apache Kafka, Schema Evolution, Data Pipeline, Event Sourcing, Real-Time Data, Event Stream Enrichment

    Issue

    Vol. 2 No. 2 (2019)

    Section

    Articles

    How to Cite

    [1]
    T. Nkosi, “Continuous Data Transformation in Event-Driven Microservices”, IJAIDT, vol. 2, no. 2, pp. 01–14, Aug. 2019, doi: 10.67228/30713315/IJAIDT-2019PII0Q75.

  • Abstract

    In modern distributed systems, event-driven microservices have emerged as a robust architectural paradigm, offering scalability, resilience, and decoupled communication. However, these systems often require real-time or near-real-time transformation of data across services and domains. Continuous data transformation—the ongoing process of modifying, enriching, or aggregating event data as it flows through the system—is critical for maintaining data consistency, enabling business insights, and supporting downstream consumers. This paper explores architectural patterns, technologies, and best practices for implementing continuous data transformation in event-driven microservices. It highlights common challenges such as schema evolution, message format variability, stateful processing, and system observability. Furthermore, it presents real-world use cases, compares toolsets such as Apache Kafka Streams, Apache Flink, Debezium, and AWS Kinesis, and proposes a reference architecture to guide practitioners in designing scalable transformation pipelines.

  • References

    [1] Kreps, J. (2014). I Heart Logs: Event Data, Stream Processing, and Data Integration. O’Reilly Media.

    [2] Kleppmann, M. (2017). Designing Data-Intensive Applications. O’Reilly Media.

    [3] Akidau, T., & Chernyak, S. (2018). Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing. O’Reilly Media.

    [4] Karau, H., & Warren, R. (2017). High Performance Spark: Best Practices for Scaling and Optimizing Apache Spark. O’Reilly Media.

    [5] Narkhede, N., Shapira, G., & Palino, T. (2017). Kafka: The Definitive Guide. O’Reilly Media.

    [6] Fowler, M. (2019). "Event-Driven Architecture." https://martinfowler.com/articles/201701-event-driven.html

    [7] Hassan, S., Bahsoon, R., & Kazman, R. (2019). Microservice transition and its granularity problem: A systematic mapping study. arXiv preprint arXiv:1903.11665.

    [8] Kleppmann, M. (2017). Designing Data-Intensive Applications. O’Reilly Media.

    [9] Newman, S. (2015). Building Microservices: Designing Fine-Grained Systems. O’Reilly Media.

    [10] Richardson, C. (2018). Microservices Patterns: With examples in Java. Manning Publications.

    [11] Hohpe, G., & Woolf, B. (2004). Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley.

    [12] Fowler, M., & Lewis, J. (2014). Microservices: A definition of this new architectural term. ThoughtWorks.

    [13] Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. Proceedings of NetDB.

    [14] Carbone, P., Katsifodimos, A., Ewen, S., et al. (2015). Apache Flink: Stream and batch processing in a single engine. IEEE Data Engineering Bulletin.

    [15] Akidau, T., Chernyak, S., & Lax, R. (2018). Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing. O’Reilly Media.

    [16] Stonebraker, M., Çetintemel, U., & Zdonik, S. (2005). The 8 requirements of real-time stream processing. ACM SIGMOD Record.

    [17] Gorton, I., & Klein, J. (2014). Distribution, data, deployment: Software architecture convergence in big data systems. IEEE Software.

  • Downloads