Real-Time Data Harmonization Techniques for Multi-Source Integration
-
DOI:
https://doi.org/10.67228/30715717/IJDEIC-2018PI3H8LPublished 02-10-2018
Real-Time Data Integration, Data Harmonization, Multi-Source Systems, Etl, Schema Matching, Data Transformation, Stream Processing, Semantic Integration, Data Consistency Issue
Section
ArticlesHow to Cite
[1]R. K. Sharma and P. Natarajan, “Real-Time Data Harmonization Techniques for Multi-Source Integration”, IJDEIC, vol. 1, no. 1, pp. 01–14, Feb. 2018, doi: 10.67228/30715717/IJDEIC-2018PI3H8L.Abstract
Real-time data harmonization has become crucial for integrating diverse data sources such as transactional databases, IoT devices, web services, and enterprise systems. However, challenges like schema differences, varied formats, language inconsistencies, latency, and lack of interoperability make integration complex. This paper reviews key harmonization techniques, including schema matching, data transformation, entity resolution, and stream processing, along with ETL/ELT models and modern tools like Apache Kafka and Apache Storm. It also highlights semantic approaches using ontologies and metadata, and the role of middleware and service-oriented architectures in enabling real-time integration. The study classifies harmonization methods into syntactic, semantic, and temporal categories, analyzing their performance trade-offs. A layered architecture is proposed, covering data ingestion, transformation, alignment, and validation. Experimental results show improved data consistency, reduced latency, and higher integration accuracy, especially with hybrid approaches combining rule-based and probabilistic methods. The paper concludes by emphasizing the need for scalable, adaptive, and AI-driven harmonization solutions for future data ecosystems.
References
[1] W. H. Inmon, Building the Data Warehouse, 4th ed. New York, NY, USA: Wiley, 2005.
[2] R. Kimball and M. Ross, The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, 3rd ed. Hoboken, NJ, USA: Wiley, 2013.
[3] A. Doan, A. Halevy, and Z. Ives, Principles of Data Integration. San Francisco, CA, USA: Morgan Kaufmann, 2012.
[4] E. Rahm and P. A. Bernstein, “A survey of approaches to automatic schema matching,” VLDB Journal, vol. 10, no. 4, pp. 334–350, 2001.
[5] J. Euzenat and P. Shvaiko, Ontology Matching, 2nd ed. Heidelberg, Germany: Springer, 2013.
[6] M. Lenzerini, “Data integration: A theoretical perspective,” in Proc. ACM PODS, 2002, pp. 233–246.
[7] D. J. Abadi et al., “Aurora: A new model and architecture for data stream management,” VLDB Journal, vol. 12, no. 2, pp. 120–139, 2003.
[8] T. Akidau et al., “The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale data processing,” Proc. VLDB Endowment, vol. 8, no. 12, pp. 1792–1803, 2015.
[9] J. Kreps, N. Narkhede, and J. Rao, “Kafka: A distributed messaging system for log processing,” in Proc. NetDB, 2011.
[10] G. Cugola and A. Margara, “Processing flows of information: From data stream to complex event processing,” ACM Computing Surveys, vol. 44, no. 3, pp. 1–62, 2012.
[11] S. Chandrasekaran et al., “TelegraphCQ: Continuous dataflow processing for an uncertain world,” in Proc. CIDR, 2003.
[12] T. Berners-Lee, J. Hendler, and O. Lassila, “The semantic web,” Scientific American, vol. 284, no. 5, pp. 34–43, 2001.
[13] D. Fensel, Ontologies: A Silver Bullet for Knowledge Management and Electronic Commerce. Berlin, Germany: Springer, 2004.
[14] A. Gandomi and M. Haider, “Beyond the hype: Big data concepts, methods, and analytics,” International Journal of Information Management, vol. 35, no. 2, pp. 137–144, 2015.
[15] M. Stonebraker et al., “The end of an architectural era: (It’s time for a complete rewrite),” in Proc. VLDB, 2007, pp. 1150–1160.
Downloads
How to Cite
[1]R. K. Sharma and P. Natarajan, “Real-Time Data Harmonization Techniques for Multi-Source Integration”, IJDEIC, vol. 1, no. 1, pp. 01–14, Feb. 2018, doi: 10.67228/30715717/IJDEIC-2018PI3H8L.