Calculating discord using the two time series data

[adityabhandwalkar]

Hello everyone,
I would like to calculate the discord across two time series , is there any way I can do that.
Thanks

[seanlaw]

@adityabhandwalkar Welcome to the STUMPY community and thank you for your question! In general, let's say you had two time series called T1 and T2, then you can

1. concatenate T1 and T2 into one long time series
2. compute the matrix profile for this concatenated time series
3. extract the discords located at the maximum values (see this example)

import numpy as np
import stumpy

T1 = np.random.rand(10)
T2 = np.random.rand(20)
concatenated = np.concatenate([T1, [np.nan], T2])
mp = stumpy.stump(concatenated, m)

[adityabhandwalkar]

Hi Sean,

Thanks for your answer, So Let me just put it in another way , https://stumpy.readthedocs.io/en/latest/Tutorial_AB_Joins.html in this example the motif or conserved pattern is detected where queen_motif_index = queen_mp[:, 0].argmin() so I wanted to ask you instead of queen_motif_index = queen_mp[:, 0].argmin() if we could take queen_motif_index = queen_mp[:, 0].argmax() can we get discord using it.

[seanlaw]

if we could take queen_motif_index = queen_mp[:, 0].argmax() can we get discord using it.

Yes, you can.

[adityabhandwalkar]

Thanks,

Adding to this further , What I would like to do is that to cluster these euclidean distances which we get from the matrix , is there any way I can cluster using hdbscan .

[[31.391212081788165 467 -1 467]
 [31.38150860900448 468 -1 468]
 [31.383205716915594 469 -1 469]
 ...
 [34.2981229015963 45385 45385 -1]
 [34.29166559828466 45386 45386 -1]
 [34.30754494100886 45387 45387 -1]]

[seanlaw]

Note that STUMPY gives you a matrix profile in the first column and not the full pairwise "distance matrix" (there is no way to get this in STUMPY), which is typically needed for clustering with hdbscan. Given that you only have one single matrix profile, I'm not sure that it is meaningful to perform clustering here as I wouldn't know how to interpret or make sense of the results.

Having said that, if you are looking to cluster multiple different time series based on how similar their subsequences may be then you may find some clues in this related question.

[adityabhandwalkar]

ok so here is what I am doing
I was using the single mp with the first(0) and second(1) column as feature and index below resp.
and I got the results like this, So is this a correct way I could cluster them

         mp = stumpy.stump(y, m= int(window_size))
        # using the euclidean distance as feature 
        cd = mp[:, [0, 1]]

    def clustering(self,path):
          
        file_read = TdmsFile.read(path)
        data = file_read.groups()[0].channels()[0].data
        y= fft(data)
        sample = len(data)
        N = 96000 # sampling rate
        frequency = np.linspace (0.0, sample, int (N/2))
        y= 2/N * (y[0:np.int (N/2)]).astype(np.float64)
        #96000  sampling rate   but changed into frequency domain
        # matrix profile generation
        window_size = input('Enter the window size?\n')
        mp = stumpy.stump(y, m= int(window_size))
        # using the euclidean distance as feature 
        cd = mp[:, [0, 1]]
        df = pd.DataFrame(cd)
        nbrs = NearestNeighbors(n_neighbors=5).fit(df)
        # Find the k-neighbors of a point
        neigh_dist, neigh_ind = nbrs.kneighbors(df)
        # sort the neighbor distances (lengths to points) in ascending order
        # axis = 0 represents sort along first axis i.e. sort along row
        sort_neigh_dist = np.sort(neigh_dist, axis=0)
        k_dist = sort_neigh_dist[:, 2]
        clusters = DBSCAN(eps=15, min_samples=4 ).fit(df)
        labels = clusters.labels_
# get cluster labels

        plt.scatter(df.iloc[:,1], df.iloc[:,0], c = labels,cmap='magma') # plotting the clusters
        plt.xlabel("index") # X-axis label
        plt.ylabel("feature") # Y-axis label
        plt.show()

[seanlaw]

I was using the single mp with the first(0) and second(1) column as feature and index below resp.
and I got the results like this, So is this a correct way I could cluster them

Unfortunately, I don't know. As I mentioned before, I wouldn't know how to interpret the results of trying to cluster a single matrix profile directly. In other words, I wouldn't know the meaning of the clustering results. Therefore, I cannot comment on whether or not this is "correct" or whether it is meaningless.

[adityabhandwalkar]

Ok, So If i were to calculate the pairwise distance between two time series and write distance matrix what Should I do,
lets say time_series1 and time_series2 , how can I get the distance matrix pair wise?

[seanlaw]

As mentioned before, it is not possible to output the pairwise distance matrix in STUMPY due to memory limitations for storing a full n x n matrix. You may be able to do it using the cdist function in scipy (see docs) but one must take care to z-normalize each subsequence where appropriate. However, this is beyond the scope of STUMPY support.

[adityabhandwalkar]

Can we apply the DAMP(Discord Aware Matrix Profile) with stumpy for Forward Processing and Backward Processing as in
[(https://dl.acm.org/doi/pdf/10.1145/3534678.3539271)]

[seanlaw]

We currently have an open issue to explore DAMP but we have other higher priority items that need to be completed first. Please feel free to follow that issue as it progresses.

[adityabhandwalkar]

Issue resolved