Suggestions on how to detect anomalies in around 200k time series, maybe with deep learning, maybe in a fast way?

I have tried Time Series Anomaly Detection Using Deep Learning, with just a few thousands of time series, but it is very slow...

Any suggestion to get anomalies detected with 200k time series in a faster way?

Answer 1

[Eamonn]

Hello

The DAMP algorithm (in Matlab) can do anomlay detecion at 100,000+ Hz

It is the only time series anomlay detecion algorthm to process a dataset with one trillion datapoints.

OH, and it beat deep learning on all the benchmarks.

It will be here https://www.cs.ucr.edu/~eamonn/MatrixProfile.html next week, but in the meantime, if you want the code/paper, I am happy to share it.

[Sim]

Thanks a lot @Eamonn!

About paper and code, absolutely Yes, please!

I guess you need my email, since I do not know if both paper and code could be here attached, in the MATLAB Answers...

I would have some comments in this regard, that I could summarise here as bullet points :-)

• Related to the Matrix Profile methodology (as far as I have understood, it is based on the calculation of eucledian distances among all the possible subsuquences of two time series), I have only heard about stumpy, but maybe the mechanism is the same / very similar... So, if I have understood correctly, the Matrix Profile will be calculated for all the possible couples of time series, that, in my case, are around 200k (and all of them contain 24 data, corresponding to 24 hours in a day)....
• Do you also have a paper (or a pptx presentation) related to the differences btw DAMP and deep learning ? Why is DAMP better or more efficient / robust than deep learining ? It would be really nice to see, and maybe cite, related papers :-) ....Now I am also wondering if there is a review paper (that I did not find so far) about all, or most of, the algorithms for detecting anomalies in time series, with a discussion on the comparison about them!
• In this period I am also trying the BEAST: A Bayesian Ensemble Algorithm for Change-Point Detection and Time Series Decomposition algorithm from Zhao et a. (2019) Detecting change-point, trend, and seasonality in satellite time series data to track abrupt changes and nonlinear dynamics: A Bayesian ensemble algorithm. I am then wondering if the anomalies detected with the DAMP algorithm (in Matlab), could be related to the abrupt changes found with BEAST.. this would be very interesting

Sorry, too many comments, but I am really curious about DAMP!

[Eamonn]

Hello

Stumpy is just a third party implenation of the Matrix Profile.

There is a paper on DAMP here: https://www.cs.ucr.edu/~eamonn/DAMP_long_version.pdf

There is linked matlab code (from the paper), however do check back in a week when the code will be update.

Unfortunately, 95% of all the claims about deep learnign anomlay detection are wrong! Here is a talk to explain why https://www.youtube.com/watch?v=Vg1p3DouX8w

Irrational Exuberance Why we should not believe 95% of papers on Time Series Anomaly Detection

[Sim]

Thanks a lot @Eamonn !

I have read the paper you have suggested and I have watched the youtube video: they are both very very interesting and promising! Given the benchamark tests I would prefer DAMP than deep learning methods, since (way much) simpler, and at the same time, accurate, robust and faster than deep learning... this is what I have understood from that paper (and others ones).

I have already tried DAMP on my machine, with one time series here attached

ts.mat

However, I was not able to use all the DAMP variants. Before accepting the Answer, I would like to make all the DAMP functions work...

It looks like that the function DAMP works....

load('/Users/.../ts.mat', 'w3')

>> DAMP(w3)
Pruning Rate: 0
Predicted discord score/position: 4.017/1021

...as well as the DAMP_X_Lag_Amnesic function...

>> DAMP_X_Lag_Amnesic(w3)
Pruning Rate: 0
Predicted discord score/position: 4.017/1021

...as well as the DAMP_fullMP function....

>> DAMP_fullMP(w3);
Pruning Rate: 0.9274
Predicted discord score/position: 4.9389/844

...but, unfortunately, the DAMP_topK function gives me an error:

>> K = 5

K =

     5

>> DAMP_topK(w3,K)
Index exceeds the number of array elements. Index must not exceed 1056.

Error in DAMP_topK (line 26)
        Left_MP(i) = min( MASS_V2(T(1:i), T(i:i+SubsequenceLength-1)));

Do you know how I could solve that error ?

Again, many thanks for informing me about this algorithm!

[Eamonn]

Hello

Thanks for using DAMP.

We are going to push a new cleaner version of the code to the website in the next 24 hours.(with a nice how to)

In the meantime, if you have any detailed questions..

1) About the code, email Yue

2) About the underling logic of anomaly detection, you can email me.

I cant seem to download ts.mat, if you want, you can email it to me.

eamonn

[Sim]

Thanks @Eamonn, I think I will drop you an email later...

Meanwhile, just for a sake of completeness, I would like to inform you that I might have found "a way" to overtake the error coming out when using DAMP_topK. Indeed, I changed

CurrentIndex = 1000+1; % line 9 of DAMP_topK

into

CurrentIndex = 24;

getting this figure

However, if I change that parameter into

CurrentIndex = 50;

I get the following:

By changing CurrentIndex, the predicted scores/positions change a bit (i.e. the first two "peaks" in the previous case, i.e. with CurrentIndex = 24 disappear and, obviously, two "lower peaks" appear with CurrentIndex = 50, in order to always have K = 5)...

Therefore, what is the right value for the variable CurrentIndex ?

[Sim]

@Eamonn, email just sent to you :-)

P.S.: just in case I attach here again the ts.mat file (including both "w3" and "w4" time series)

Answer 2

[Image Analyst]

Are you talking about training or prediction?

For what it's worth, here is another anomaly detection demo from Mathworks.

https://www.mathworks.com/videos/anomaly-detection-for-industrial-processes-and-machinery-with-matlab-1648135588169.html?s_tid=srchtitle

Anomaly Detection for Industrial Processes and Machinery with MATLAB Video

Learn how to apply statistical and machine learning based anomaly detection techniques to industrial processes and machinery.

www.mathworks.com

[Sim]

Thanks a lot @Image Analyst, very kind! I will try the algorithm you have kindly suggested and let's see the differences with Time Series Anomaly Detection Using Deep Learning... Thanks a lot! :)

[Image Analyst]

Well of course you could always get a faster computer with more memory. Get the best video adapter you can afford with the most GPU memory you can afford. And get the Parallel Processing Toolbox, which will speed it up quite a bit (if you end up having to use the CPU because your GPU does not have enough memory and you get an "out of memory" error). The GPU does not know how to use disk space to allocate additional memory (everything happens on the GPU) but the CPU can get as much virtual memory from the disk drive as it needs (or that you allow).

[Sim]

Thanks a lot @Image Analyst, in this moment I do not have access to faster machines, but I could try with the parallel processing toolbox. However, by considering the paper/s showed by @Eamonn, it looks like that (for time series) deep learning is outperformed by another algorithm based on the Matrix Profile :-)