Anomaly detection

[codenicks]

Hi, I am trying to use stumpy for univariate timeseries anomaly detection. I am still not sure if only discords represent anomalies or motifs too. I have a dataset like taxy example. Here is my discord discovery. I would like to know how to interpret this data and how I could get the index of the anomalies to plot them over the dataset with a different color. Any help is appreciated. Thankyou

!

[seanlaw]

@codenicks Welcome to the STUMPY community and thank you for your question. Are you able to provide the raw data and code for reproducing the plots above? That would really help

[codenicks]

I have watched the video now, I also found about AB-Joins, maybe this is what I am looking for, I am not sure. Considering my data I think normal matrix profile without AB-Joins should also work. First thing that I would like to know is why that gray rectangle is not overlapping the graph. Also considering I found the top discord, how should I find the others ? Thankyou and I am sorry for so many questions.

[seanlaw]

First thing that I would like to know is why that gray rectangle is not overlapping the graph.

Ahh, it's because of rect = Rectangle((discord_idx, 0), m, 40, facecolor='lightgrey'). The rectangle is drawn by specifying the bottom-left-hand corner followed by the width (m) and height (40). So, the height needs to be changed accordingly. Perhaps, this is better:

rect = Rectangle((discord_idx, 0), m, max(data), facecolor='lightgrey')

This way, the rectangle is drawn and extends all the way to the top of the plot.

Also considering I found the top discord, how should I find the others ?

Once you find the first discord, you'll likely want to ignore any further discords in this area. So, what you can do is set the matrix profile values around each discord to, say, np.NINF (negative infinity):

mp = stumpy.stump(data, m)
discord_idx = np.argsort(mp[:, 0])[-1]
# Do something with this discord

# set last discord values to np.NINF
from stumpy import config
excl_zone = int(np.ceil(m / config.STUMPY_EXCL_ZONE_DENOM))
zone_start = max(0, discord_idx - excl_zone)
zone_stop = min(mp[:, 0].shape[-1], discord_idx + excl_zone)
mp[zone_start : zone_stop + 1, 0] = np.NINF

# Find new discord just like before
discord_idx = np.argsort(mp[:, 0])[-1]
# Do something with this discord

# Repeat as needed

[codenicks]

For the rectangle I get this error if I do max(data)

Regarding the discords, is there a way to do a for loop until I get all discords ?

[seanlaw]

For the rectangle I get this error if I do max(data)

I don't think your code is equivalent to max(data) (where data is your 1-D input to stumpy.stump). Based on line 4 of your code, the equivalent would be:

rect = Rectangle((discord_idx, 0), m, max(test_anomaly['T-HEX']), facecolor='lightgrey')

Regarding the discords, is there a way to do a for loop until I get all discords ?

Yes, you'd do as I proposed above with the "repeat":

from stumpy import config

excl_zone = int(np.ceil(m / config.STUMPY_EXCL_ZONE_DENOM))
mp = stumpy.stump(data, m)
discord_indices = []

# Get top 10 discords
for k in range(10):
    discord_idx = np.argsort(mp[:, 0])[-1]
    if mp[discord_idx, 0] == np.NINF:
        break

    # Do something with this discord
    discord_indices.append(discord_idx)

    # set discord values to np.NINF
    zone_start = max(0, discord_idx - excl_zone)
    zone_stop = min(mp[:, 0].shape[-1], discord_idx + excl_zone)
    mp[zone_start : zone_stop + 1, 0] = np.NINF

Something like that

[codenicks]

It is working, thankyou very much !