I started working on anomaly detection applications in 2015. It was my first assignment outside of Finland, and I worked as a consultant for one of the most well-known toy manufacturing companies. It has been a great learning experience for me to see how anomaly detection has been applied in various industries, and also to see how the methods have evolved from classical methods such as Hotelling T2 square to advanced methods such as autoencoder neural networks

Anomaly detection

Anomaly detection is a critical task across a wide range of industries, from manufacturing and industrial processes to finance and healthcare. Anomaly detection refers to the identification of rare or unusual events or observations that differ from the expected or typical behavior.

One of the primary applications of anomaly detection is fault detection. In manufacturing and industrial processes, for example, anomalies in sensor data can indicate equipment failures, product defects, or other issues that can lead to costly downtime, safety risks, and other problems.

In the finance industry, anomaly detection can be used to detect fraud, such as unusual transactions or patterns of activity that deviate from the norm. It can also be used to identify potential market anomalies, such as sudden spikes or dips in stock prices, that may indicate opportunities for investment or risk management.

In healthcare, anomaly detection can help identify anomalies in medical images or other data that could indicate the presence of a disease or condition. It can also be used in predictive maintenance of medical equipment to detect potential failures and reduce downtime.

Unsupervised approach with Autoencoder

Autoencoder is a popular deep learning technique that is widely used in anomaly detection for business applications. In essence, it is an unsupervised learning method that trains a neural network to learn the patterns and relationships in a given dataset, and then uses this knowledge to detect any outliers or anomalies in new data. Autoencoders can learn a compressed representation of the input data, and they work by minimizing the difference between the input and the output. In anomaly detection, the input is considered normal data, and any deviation from this data results in a high reconstruction error, which indicates an anomaly.

The above figure shows an example how an autoencoder can identify anomaly events of a system just via a single metric (the peaks marked with red dots). Autoencoders are powerful as they can learn complex representations of the data, which allows them to detect anomalies that are not captured by traditional statistical methods.

Hotelling's T2 is a classical method for anomaly detection which also uses a single metric (called Hotelling T2 score) to detect anomaly events. This metric is calculated from multiple variables. However, Hotelling T2 score is too sensitive to the input data. The below figure shows the Hotelling T2 score calculated from the same dataset with the one used for the autoencoder above. The multimodal distribution of the Hotelling T2 score is the result from the fact that there are some input variables having multimodal distribution. It is almost impossible to identify anomaly events from this Hotelling T2 score.

Anomaly detection is crucial for identifying potential issues and threats in various industries, and autoencoders and and its more advanced version, variational autoencoder, are powerful tools for detecting anomalies and predicting future trends. As businesses continue to collect and analyze more data, the importance of anomaly detection and the use of advanced methods like autoencoders and VAEs will only increase.