I want to terminate a cuda stream if it runs too slow.
I was looking at the cuda manual and I found cudaStreamDestroy.
It says that cudaStreamDestroy will “Destroys and cleans up the asynchronous stream”.
Does that mean even if a cuda stream is running, I can terminate it and start running some other cuda stream?
Or is there anything in CUDA similar to pthread’s “pthread_cancel”, “pthread_kill”, and etc?
Not that I am aware of, other than destroying the context in which the kernel is running. Even then it might not be “graceful” and certainly on older versions of CUDA it was possible to leave the GPU or driver in a state which might require a reboot to clear. Much better to design a kernel that has a well defined upper bound on runtime in the first place.
I see, so there is a way to terminate the kernel but not what I wanted (I want to run another kernel in the same context). There is no such api like “pthread_cancel” or “pthread_kill” in pthread. I am just curious about the reason. Is it because current runtime implementation can’t support it? Or it is just not there yet but this feature can be added ? Besides, is the function to destroy the context “cuCtxDestroy”?
Thanks. This is one way to terminate the kernels by themselves if the kernels constantly check the flag. For async memory transfers in different kernel streams, I suppose it is still not possible to terminate the transfer?
I have tried to kill a kernel that is running by using cudaDeviceReset, cuCtxDestroy, and cuDevicePrimaryCtxReset. It seems that all of them wait for the kernel to finish and then
they probably release GPU resources. Is there any other solution? Is there an explanation?