Self-Healing Data Pipelines: Leveraging AI for Fault Detection and Resolution in Real-Time Analytics Systems

  • Authors

    • Priya Kapoor Business Analyst, Accenture, India. Author

    DOI:

    https://doi.org/10.67228/30715717/IJDEIC-2021PII6R2F

    Published 11-18-2021

  • Self-Healing Data Pipelines, Real-Time Analytics, Fault Detection, Fault Resolution, Artificial Intelligence, Anomaly Detection, Root Cause Analysis, Automated Recovery, Data Pipeline Monitoring, Predictive Maintenance

    Issue

    Vol. 4 No. 2 (2021)

    Section

    Articles

    How to Cite

    [1]
    P. Kapoor, “Self-Healing Data Pipelines: Leveraging AI for Fault Detection and Resolution in Real-Time Analytics Systems”, IJDEIC, vol. 4, no. 2, pp. 01–10, Nov. 2021, doi: 10.67228/30715717/IJDEIC-2021PII6R2F.

  • Abstract

    Real-time analytics systems rely heavily on complex data pipelines to process and deliver insights promptly. However, these pipelines are prone to faults that can disrupt data flow, leading to inaccurate analytics and operational downtime. This paper explores the concept of self-healing data pipelines empowered by artificial intelligence (AI) techniques to detect, diagnose, and resolve faults autonomously. We present an architecture that integrates AI-driven anomaly detection, root cause analysis, and dynamic fault resolution within real-time analytics systems. Our approach aims to enhance pipeline resilience, reduce manual intervention, and improve overall system reliability. Experimental evaluation demonstrates that AI-enabled self-healing mechanisms significantly reduce downtime and maintain data integrity in dynamic environments. We also discuss the challenges and future research directions in deploying such systems at scale.

  • References

    [1] Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer, 36(1), 41–50. https://doi.org/10.1109/MC.2003.1160055

    [2] Ganek, A. G., & Corbi, T. A. (2003). The dawning of the autonomic computing era. IBM Systems Journal, 42(1), 5–18. https://doi.org/10.1147/sj.421.0005

    [3] Dudley, G., Joshi, N., Ogle, D. M., Subramanian, B., & Topol, B. B. (2004). Automatic self-healing systems in a cross-product IT environment. In Proceedings of the International Conference on Autonomic Computing (ICAC 2004) (pp. 210–217). IEEE. https://doi.org/10.1109/ICAC.2004.1301393

    [4] Sterritt, R., Bustard, D. W., & McCrea, A. (2003). Autonomic computing correlation for fault management system evolution. In Proceedings of the IEEE International Conference on Industrial Informatics (pp. 149–154). IEEE. https://doi.org/10.1109/INDIN.2003.1300275

    [5] Ghosh, D., Sharman, R., Rao, H. R., & Upadhyaya, S. (2007). Self-healing systems—Survey and synthesis. Decision Support Systems, 42(4), 2164–2185. https://doi.org/10.1016/j.dss.2006.06.011

    [6] Sterritt, R., & Bustard, D. W. (2004). Autonomic networks: Engineering the self-healing property. Engineering Applications of Artificial Intelligence, 17(7), 727–739. https://doi.org/10.1016/j.engappai.2004.08.028

    [7] Schneider, C., Barker, A. D., & Dobson, S. A. (2014). Autonomous fault detection in self-healing systems: Comparing Hidden Markov Models and artificial neural networks. In Proceedings of the 4th International Workshop on Adaptive Self-Tuning Computing Systems (ADAPT ’14) (pp. 24–31). ACM. https://doi.org/10.1145/2553062.2553065

    [8] Schneider, C., Barker, A. D., & Dobson, S. A. (2014). Autonomous fault detection in self-healing systems using Restricted Boltzmann Machines. In Proceedings of the 11th IEEE International Conference and Workshops on Engineering of Autonomic and Autonomous Systems (EASe 2014). IEEE.

    [9] Krupitzer, C., Roth, F. M., VanSyckel, S., Schiele, G., & Becker, C. (2015). A survey on engineering approaches for self-adaptive systems. Pervasive and Mobile Computing, 17, 184–206. https://doi.org/10.1016/j.pmcj.2014.09.009

    [10] Dundar, B., Astekin, M., & Aktas, M. S. (2016). A big data processing framework for self-healing Internet of Things applications. In 2016 12th International Conference on Semantics, Knowledge and Grids (SKG) (pp. 62–68). IEEE.

  • Downloads