Unified Data Lake Architectures: Integrating Object-to-Stream Ingestion, Server less Kafka Streaming ETL, and Near Real-Time Analytical Indexing

Abstract

Enterprise data platforms often run into a common operational dilemma: business stakeholders require sub-second operational log search, while data engineering and BI teams require massive, multi-terabyte analytical queries over historical data. Building separate streaming and batch infrastructure leads to duplicate codebases, out-of-order event drift, and excessive cloud infrastructure costs. In this paper, we describe a unified, production-tested cloud data lake architecture that solves these challenges on AWS. Instead of running periodic, heavy SQL queries against production transactional databases, we repurpose AWS Database Migration Service (DMS) as a permanent, non-blocking Change Data Capture (CDC) streaming source that captures row-level binary log deltas and pushes them directly into Amazon Kinesis Data Streams and Amazon S3. For downstream analytical processing, we implement an adaptive serverless Apache Spark ETL pipeline using AWS Glue to compact micro-batches into 256 MB Snappy-compressed Apache Parsquet partitions, resolving the common 'small file problem' on object storage. In parallel, operational server access logs are routed via event-driven AWS Lambda consumers to Amazon Elasticsearch Service (7.x) using a tri-tier index lifecycle management strategy. We evaluated this architecture across real production enterprise workloads sustaining over 100,000 transactions per second (TPS). Our benchmarks show a 78.4% reduction in median data ingestion latency (p99 < 4.2s), an 82.8% reduction in S3 storage footprints via columnar compression, and a 19x speedup in distributed SQL query execution compared to raw object scans.