Anomalies often referred to as outliers, are data points or patterns in data that do not conform to a notion of normal behavior. Automatically detecting and correctly classifying something unseen as anomalous is a challenging problem that has been tackled in many different manners over the years. Traditional machine learning approaches are sub-optimal when it comes to high dimensional data because they fail to capture the complex structure in the data. This is where deep learning methods can be leveraged for the task.
This talk reviews a set of relevant deep learning model architectures including autoencoders, variational auto-encoders, generative adversarial networks, and sequence-to-sequence methods, and addresses how they can be applied to the task of anomaly detection, comparing them in terms of training, inference, and storage costs. Anomaly detection using each of these models is explored as a function of how they can be applied to first model normal behavior, and then this knowledge is exploited to identify deviations.
In addition, we provide practical guidance for the successful implementation of anomaly detection systems within enterprises across key metrics like interpretability, reduction of false positives and scalability. We show benchmark results from applying them using several datasets and discuss industry best practices for deploying each model. Finally, we provide intuition for how and why these algorithms work by demoing them with a working prototype.
Overview and Author Bio
Deep Learning for Anomaly Detection
Research Engineer | Cloudera Fast Forward Labs