A Comparative Analysis of Anomaly Detection from Microservice Generated Unstructured Logs

Abstract

In a process, to ensure increased reliability and better availability, it is very important to detect any anomalies that refer to any abnormality observed in the behaviour of a standard process. The breakdown of service(s) eventually leads to production loss, and at the same time, a system that is unreliable brings lots of challenges to the operations team.  Anomaly detection plays a significant role to ensure that an application is reliable, secured and available for user requests. For the overall performance optimization of a cloud microservice based application without any disruption in service, and identification of possible security threat it is much essential that the anomalies must be detected and responded to, in time. In real life large microservice based production infrastructures environments, even though ample instance of normal activities is available, it is not possible to predict and create a dataset of anomalies. So these kind of data are not suitable for a supervised two-class classification. In this work, unsupervised one-class approaches Local Outlier Factor, Isolation Forest and One Class SVM are used to find anomalies. On experimentation these models have obtained a high accuracy of 98% to 99%. On comparing the performance of the models, One-Class SVM is found to produce significantly higher number of False Positives in comparison to other two models.