Dask Client Trying to Access GPUs in stumpy.stumped

[geoHeil]

I try to execute https://stumpy.readthedocs.io/en/latest/ on dask (without the GPU):

import os
#os.environ["CUDA_VISIBLE_DEVICES"] = "1"
import numpy as np
import stumpy


from dask.distributed import Client
client = Client()
your_time_series = np.random.rand(100)
window_size = 50  # Approximately, how many data points might be found in a pattern

matrix_profile = stumpy.stumped(client, your_time_series, m=window_size)

But this fails with:

 CudaAPIError: [2] Call to cuDevicePrimaryCtxRetain results in CUDA_ERROR_OUT_OF_MEMORY

However, when:

os.environ["CUDA_VISIBLE_DEVICES"] = "1" 

is set it works (note it is a multi GPU machine and indeed GPU 0 is currently busy.

But my point is:

• I am using the non GPU version of stumpy: so why is it rying to use the GPU at all?
• why does this variable have no effect unless it is set prior to starting the dask cluster?

[seanlaw]

This is strange indeed and I wonder if this is more of a Dask problem rather than a STUMPY problem. And you are absolutely right that stumpy.stumped does not use GPUs and we have no support for GPUs in that function (GPUs are supported in the gpu_stump function). What happens if you run:

import numpy as np
import stumpy


from dask.distributed import Client
client = Client()
your_time_series = np.random.rand(100)
window_size = 50  # Approximately, how many data points might be found in a pattern

matrix_profile = stumpy.stump(your_time_series, m=window_size)  # Notice this is `stump` and not `stumped`

Internally, stumped simply breaks down your time series into smaller chunks and then passes the data to stump. Neither should be using GPUs or need any knowledge of it.

Are you able to provide a full traceback?

[geoHeil]

This works just fine.

[seanlaw]

@geoHeil Would you mind providing a full traceback?

[seanlaw]

@geoHeil Any updates on this or do you think we can close it for now?

[geoHeil]

Interesting: this bug has to do with the GPU utilization. Today, the 1st gpu is empty and I do not have any problems at all starting this task.

I will close it for now.