Automated Data Transformation Using Intelligent Rule-Based Systems

  • Authors

    • Tom DeMarco Software Engineering Consultant, Atlantic Systems Guild, United States. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2024PII7D1M

    Published 11-09-2024

  • Automated Data Transformation, Rule-Based Systems, Etl Automation, Data Integration, Intelligent Systems, Data Processing, Semantic Mapping

    Issue

    Vol. 7 No. 2 (2024)

    Section

    Articles

    How to Cite

    [1]
    T. DeMarco, “Automated Data Transformation Using Intelligent Rule-Based Systems”, IJDEIC, vol. 7, no. 2, pp. 01–12, Nov. 2024, doi: 10.67228/30715717/IJDEIC-2024PII7D1M.

  • Abstract

    This paper explores automated data transformation using intelligent rule-based systems to address the limitations of traditional ETL processes, which are often manual, rigid, and error-prone. The proposed approach integrates rule-based reasoning with metadata-driven transformation to handle complex data from heterogeneous sources such as structured, semi-structured, and unstructured data. The system features a modular architecture including data ingestion, rule definition, execution, and validation. It applies condition–action rules for tasks like normalization, filtering, aggregation, and enrichment, along with a feedback mechanism for continuous improvement. Experimental results show that the approach significantly reduces transformation time while maintaining accuracy and improving data quality. Challenges such as rule conflicts, scalability, and legacy integration are also addressed through strategies like rule prioritization and hybrid architectures. The study concludes that intelligent rule-based systems offer a scalable and efficient solution for modern data transformation in big data and real-time environments.

  • References

    [1] Kimball, R., & Caserta, J. (2011). The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, Conforming, and Delivering Data. Wiley.

    [2] Vassiliadis, P. (2009). “A Survey of Extract–Transform–Load Technology.” International Journal of Data Warehousing and Mining, 5(3), 1–27.

    [3] Golfarelli, M., & Rizzi, S. (2009). Data Warehouse Design: Modern Principles and Methodologies. McGraw-Hill.

    [4] Inmon, W. H. (2005). Building the Data Warehouse (4th ed.). Wiley.

    [5] Rahm, E., & Do, H. H. (2000). “Data Cleaning: Problems and Current Approaches.” IEEE Data Engineering Bulletin, 23(4), 3–13.

    [6] Wiederhold, G. (1992). “Mediators in the Architecture of Future Information Systems.” IEEE Computer, 25(3), 38–49.

    [7] Russell, S., & Norvig, P. (2021). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.

    [8] Jackson, P. (1998). Introduction to Expert Systems (3rd ed.). Addison-Wesley.

    [9] Giarratano, J., & Riley, G. (2005). Expert Systems: Principles and Programming (4th ed.). Thomson.

    [10] Domingos, P. (2012). “A Few Useful Things to Know About Machine Learning.” Communications of the ACM, 55(10), 78–87.

    [11] Stonebraker, M., et al. (2018). “Data Curation at Scale: The Data Tamer System.” CIDR Conference.

    [12] Zaharia, M., et al. (2016). “Apache Spark: A Unified Engine for Big Data Processing.” Communications of the ACM, 59(11), 56–65.

    [13] Krishnan, S., et al. (2016). “ActiveClean: Interactive Data Cleaning for Statistical Modeling.” Proceedings of the VLDB Endowment, 9(12), 948–959.

    [14] Rekatsinas, T., et al. (2017). “Holoclean: Holistic Data Repairs with Probabilistic Inference.” VLDB, 10(11), 1190–1201.

    [15] Abedjan, Z., Golab, L., & Naumann, F. (2016). “Profiling Relational Data: A Survey.” VLDB Journal, 24(4), 557–581.

    [16] Gajula, S. (2023). A review of anomaly identification in finance frauds using machine learning system. International Journal of Current Engineering and Technology, 13(6), 568–575. https://ijcet.evegenis.org/index.php/ijcet/article/view/820

  • Downloads