Anomaly Detector & Visualiser

Modified on Wed, 1 May, 2024 at 8:41 PM

Description

Applies an anomaly detection model to a data set and returns the anomaly score for each channel in a raw and aggregated way.

Application

Engineers often accumulates a large amount of test data that is necessary to develop a new product. However, the data needs to be trusted to be used, and sometimes it can be anomalous, whether because the whole systems is behaving in a new unexpected way, or because the test instrumentation is faulty (e.g. failing sensors). This step, combined with Anomaly Detection Model can enable a user have a quick and complete overview of the quality of their data, before using them further.

How to use

To use this step you need a trained model and a data set in which you want to detect anomalies. The data should satisfy the same requirements needed to train Anomaly Detection Models (see documentation).

Once the model and the data are selected, there is one paramter that needs to be defined:

Edge trimming

Defines which ratio of each test (in % of steps) should be ignored on the edges when computing the aggregate score. This can be used to remove noise due to edge effects.

Once the step is completed, the step Anomaly Detection Visualiser can be used with the two outputed datasets from the anomaly detector. The visualiser step enables to consume the results in a user-friendly way:

A heatmap will show the aggregated values for each combination of test (vertical axis) and channel (horizontal axis). The aggregation can be done by taking the mean anomaly score (highlighting prevalent anomalies) or the max anomaly score (highlighting critical anomalie). These two aggregation methods can return very different heat maps.

Any cell of the heatmap can be clicked on. This will display the measured signal of the test-channel combination, as well as the expected (reconstructed) signal and the anomaly signal.

This user interface enables to easily and intuitively navigates in the dataset and quickly focus on the existing anomalies.

Example

Here is an example where a sensor (car acceleration Y) dropout was highlighed in the heatmap (see blue cross) and displayed in the line plot (see region with high anomaly score).

In the same example, aggregating by prevalendance mainly highlight one anomaly (see blue cross). The display of the time series shows a reverse sign in the measurement of the car angle.