how does Stump.match handle N.A in time series

[lqmike]

Dear stumpy team, stump.match is such a great function within the fast pattern search!

stumpy.match(Q_df[column1], T_df[column1])

I realized if the time series, T_df[column1], has np.nan, seems stump.match will return empty array which is not the case for stumpy.core.mass_absolute(). As long as I do T_df['column1'].dropna() it is back to normal. what is the best practice to handle this as I do not want to do dropna() on specific columns worry the returned indices may end up being inconsistent across columns. Thanks a lot! Not exactly a programer, sorry for the simple question!

And if I could ask one more, is there any great acceleration method if I want to search large amount of patterns and find the stumpy.match. Thank you!

Mike

[seanlaw]

@lqmike Thank you for your question. Can you please execute Q_df[column1].to_csv("Q.csv") and T_df[column1].to_csv("T.csv") and upload both CSV files here (you can drag and drop them into a comment box below).

Additionally, what version of STUMPY are you using? This can be ascertained with:

import stumpy
print(stumpy.__version__)

The current version is 1.9.2 so I would first recommend upgrading to the latest version

Finally, stumpy.core.mass_absolute implies that there is no z-normalization on each of your subsequences. This is rare but, furthermore, this command has been deprecated and you should use stumpy.mass(Q, T, normalize=False) instead.

In short, STUMPY generally ignores all np.nan/np.inf values (i.e., any subsequence that contains one of these values will return a np.nan as it's distance) but it does NOT drop them. Nothing is computed if any of the pair of subsequences being compared contains np.nan/np.inf as no distance can be computed.

[lqmike]

T_df.zip

Again thanks Sean for such quick reply!

• my version is latest 1.9.2 because I only discovered this match function today so I upgraded from 1.8 to 1.9.2. I was excited because match is exactly what I have been trying to achieve using the distance_profile --> sort --> eliminate gap ....
• As my time series pattern is already normalized(0-100) so absolute is perfect for me.
• I have uploaded the file, total 359735 rows in time series order, some missing values or nan in between. 'pattern' column is the column I am trying to search, value is corresponding column I am trying to estimate.

Here is my goal, and I would love to hear your advice how to do this efficiently:

1. As it is time series, so I need to search backwards then estimate current point.
   a. Always search last 10,000 points, starting point is at row 10,001, pattern_length is 100, so [10,001:10,101] is the first Q, and [0:10000] as the first T.
   b. Set max_matches = 50, I then calculate their corresponding average 'value' (['value'] column not the pattern value) in the history when those patterns occurred. Assume the average value was 0.85 from those 50 patterns, then I will record that value as new column ['value_pred'] at end of the current pattern at 10,101 as my best guess of the value, then I can compare with the actual ['value'].
2. As I am rolling this from 10,001 all the way to 359,735, I am wondering if there is any efficient way to achieve this. So end result I will have one extra column like value_pred.

Hope the long winded description is not too confusing. Thanks a lot of the help!
Mike

[seanlaw]

So, the only major difference between stumpy.match(Q, T, normalize=False) and stumpy.mass(Q, T, normalize=False) is that match will only return a "match" if the distance (between Q and subsequences in T) is less than or equal to max_distance.

stump.match will return empty array

This can only happen when all of the distances are larger than the default max_distance which is set as 2 standard deviations below mean distance (computed from the corresponding distance profile) OR the minimum value in the distance profile (whichever is larger):

max(np.mean(D) - 2 * np.std(D), np.min(D))

Note that the default max_distance used for each Q will be different and completely dependent upon the distance profile that is computed. If you want to use the same max_distance value for all queries. Currently, if either Q or the section of your matrix profile contains np.nan then the mean/stddev will produce a max_distance threshold of np.nan as well. So, instead, you'll need to explicitly provide a max_distance threshold value OR a max_distance function. To get the same behavior as stumpy.mass(Q, T, normalize=False) you'll need to do stumpy.match(Q, T, max_distance=np.inf, max_matches=50, normalize=False). Alternatively, you can provide the following max_distance function, which should allow for np.nan to exist in your time series:

def custom_max_distance(D):
    D_copy = D.copy()
    D_copy[np.isinf(D_copy)] = np.nan
    return max(np.nanmean(D_copy) - 2.0 * np.nanstd(D_copy), np.nanmin(D_copy))

stumpy.match(Q, T, max_distance=custom_max_distance, max_matches=50, normalize=False)

@mexxexx Do you think it might make sense to update the default max_distance to use np.nanmean and np.nanstd instead?

if max_distance is None:  # pragma: no cover

        def max_distance(D):
            D_copy = D.copy()
            D_copy[np.isinf(D_copy)] = np.nan
            return max(np.nanmean(D_copy) - 2.0 * np.nanstd(D_copy), np.nanmin(D_copy))

[lqmike]

Hi Sean Q_df is not a separate data, just the slice of the T_df like I mentioned length of 100 on the T_df

[mihailescum]

@seanlaw I think updating the default max_distance in the way you proposed is a good idea :thumsup: we did not consider NaNs at the time, but I don't see why whey shouldn't be allowed in the timeseries, and the user should get the behaviour he expects