Predictive Incident Management Using Observability Data in Cloud-Native Systems

  • Authors

    • Uday Allala Lead SRE Engineer, USA. Author

    DOI:

    https://doi.org/10.67228/30715636/IJETMR-2020PI2V8C

    Published 03-08-2020

  • Cloud-native Systems, Observability, Incident Management, Predictive Analytics, Anomaly Detection, Machine Learning, Incident Prediction, Distributed Systems, DevOps, AIOps

    Issue

    Vol. 3 No. 1 (2020)

    Section

    Articles

  • Abstract

    In the cloud-native environment, systems are built from distributed microservices, containers, orchestration platforms, and dynamic infrastructure, which is increasingly challenging to detect and manage incidents. In the conventional monitoring approaches, incidents are first detected when the service is degraded, and can often produce a lot of alerts, but with little context which can be used to prioritize the alerts. This paper introduces a framework for predicting incidents using multi-source observability data such as logs, metrics, distributed traces, events, service dependencies, and operational telemetry to detect patterns that precede cloud-native incidents. The framework has been designed to support the observability data pre-processing, temporal feature extraction, cross-source correlation, anomaly detection and predictive analytics to estimate the probability of emerging incidents before they can impact application services. The proposed approach also features the use of dynamic incident risk scoring and incident severity classification to prioritize incidents predicted based on their potential operational impact. Real-time observability data is correlated with historical behavioral patterns in a continuous processing pipeline, which can trigger predictive alerts to be used in proactive remediation and incident response. The framework is tested with several representative cloud-native workloads and benchmarked against traditional rule-based monitoring, traditional statistical anomaly detection and traditional machine learning based baseline detection methods through the use of accuracy of the predictions, precision, recall, f1 score, false positive rate, detection latency, and resource utilization. The results are meant to demonstrate that analysis of correlated observability data can help with earlier detection of incidents, fewer false alarms, and more efficient and accurate incident handling in dynamically-changing cloud-native deployments.

  • References

    [1] Sigelman, B. H., Barroso, L. A., Burrows, M., Stephenson, P., Plakal, M., Beaver, D., ... & Shanbhag, C. (2010). Dapper, a large-scale distributed systems tracing infrastructure.

    [2] Wang, P., Xu, J., Ma, M., Lin, W., Pan, D., Wang, Y., & Chen, P. (2018, May). Cloudranger: Root cause identification for cloud native systems. In 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID) (pp. 492-502). IEEE.

    [3] Dean, J., & Barroso, L. A. (2013). The tail at scale. Communications of the ACM, 56(2), 74-80.

    [4] Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2002, June). Pinpoint: Problem determination in large, dynamic internet services. In Proceedings international conference on dependable systems and networks (pp. 595-604). IEEE.

    [5] Sambasivan, R. R., Zheng, A. X., De Rosa, M., Krevat, E., Whitman, S., Stroucken, M., ... & Ganger, G. R. (2011). Diagnosing performance changes by comparing request flows. In 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11).

    [6] Brandón, Á., Solé, M., Huélamo, A., Solans, D., Pérez, M. S., & Muntés-Mulero, V. (2020). Graph-based root cause analysis for service-oriented and microservice architectures. Journal of Systems and Software, 159, 110432. https://doi.org/10.1016/j.jss.2019.110432

    [7] Chen, Z., Kang, Y., Gao, F., Yang, L., Sun, J., Xu, Z., ... & Lyu, M. R. (2020, February). Aiops innovations of incident management for cloud services. In National Conference on Artificial Intelligence.

    [8] Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., & Pei, D. (2019, July). Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2828-2837).

    [9] Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., ... & Zhou, R. (2019, August). Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. In Ijcai (Vol. 19, No. 7, pp. 4739-4745).

    [10] Du, M., Li, F., Zheng, G., & Srikumar, V. (2017, October). Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security (pp. 1285-1298).

    [11] Hundman, K., Constantinou, V., Laporte, C., Colwell, I., & Soderstrom, T. (2018, July). Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 387-395).

    [12] Burns, B., Beda, J., & Hightower, K. (2018). Kubernetes. Heidelberg, Germany: dpunkt.

    [13] Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). Site reliability engineering: how Google runs production systems. " O'Reilly Media, Inc.".

    [14] Salfner, F., Lenk, M., & Malek, M. (2010). A survey of online failure prediction methods. ACM Computing Surveys (CSUR), 42(3), 1-42.

    [15] Niedermaier, S., Koetter, F., Freymann, A., & Wagner, S. (2019). On observability and monitoring of distributed systems: An industry interview study. In S. Yangui, K. Drira, I. Bouassida Rodriguez, & Z. Tari (Eds.), Service-oriented computing: 17th International Conference, ICSOC 2019, Proceedings (pp. 36–52). Springer. https://doi.org/10.1007/978-3-030-33702-5_3

    [16] Zhang, Q., Zhou, C., Xiong, N., Qin, Y., Li, X., & Huang, S. (2015). Multimodel-based incident prediction and risk assessment in dynamic cybersecurity protection for industrial control systems. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 46(10), 1429-1444.

    [17] Roumani, Y., & Nwankpa, J. K. (2019). An empirical study on predicting cloud incidents. International Journal of Information Management, 47, 131–139. https://doi.org/10.1016/j.ijinfomgt.2019.01.014

    [18] Toffetti, G., Brunner, S., Blöchlinger, M., Spillner, J., & Bohnert, T. M. (2017). Self-managing cloud-native applications: Design, implementation, and experience. Future Generation Computer Systems, 72, 165-179.

    [19] Valenti, G., Lelli, M., & Cucina, D. (2010). A comparative study of models for the incident duration prediction. European Transport Research Review, 2(2), 103-111.

    [20] Beyer, B., Murphy, N. R., Rensin, D. K., Kawahara, K., & Thorne, S. (2018). The site reliability workbook: practical ways to implement SRE. " O'Reilly Media, Inc.".

    [21] Shyam, G. K., & Manvi, S. S. (2016). Virtual resource prediction in cloud environment: a Bayesian approach. Journal of Network and Computer Applications, 65, 144-154.

    [22] Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O’Reilly Media.

    [23] Allala, U. . (2019). A Zero-Trust Framework for Automated Identity Governance in Cloud-Native Applications. American International Journal of Computer Science and Technology, 1(6), 37-47. https://doi.org/10.63282/3117-5481/AIJCST-V1I6P104

  • Downloads