Is it possible to terminate a cuda stream while it is running?

Accelerated Computing CUDA CUDA Programming and Performance
1

I want to terminate a cuda stream if it runs too slow.
I was looking at the cuda manual and I found cudaStreamDestroy.
It says that cudaStreamDestroy will “Destroys and cleans up the asynchronous stream”.
Does that mean even if a cuda stream is running, I can terminate it and start running some other cuda stream?
Or is there anything in CUDA similar to pthread’s “pthread_cancel”, “pthread_kill”, and etc?

Thanks!

2

No, it will not kill a running kernel

3

Thanks!
Is there any way to kill a running kernel?

4

Not that I am aware of, other than destroying the context in which the kernel is running. Even then it might not be “graceful” and certainly on older versions of CUDA it was possible to leave the GPU or driver in a state which might require a reboot to clear. Much better to design a kernel that has a well defined upper bound on runtime in the first place.

5

I see, so there is a way to terminate the kernel but not what I wanted (I want to run another kernel in the same context). There is no such api like “pthread_cancel” or “pthread_kill” in pthread. I am just curious about the reason. Is it because current runtime implementation can’t support it? Or it is just not there yet but this feature can be added ? Besides, is the function to destroy the context “cuCtxDestroy”?

Thanks!

6

I don’t know the reason why or how/if it could be implemented. All of that is “backstage” in the driver libraries and no one other than the development team is privy to that stuff.

cuCtxDestroy() is what you would call in the driver API. If you are using the runtime API, cudaThreadExit() is what you want.

7

There’s no way to do this at the moment.

8

You may signal the threads to return by keeping a Constant memory flag that you can write to from Host code when your kernel crosses the time limit.

9

Thanks. This is one way to terminate the kernels by themselves if the kernels constantly check the flag. For async memory transfers in different kernel streams, I suppose it is still not possible to terminate the transfer?

10

Good evening

I have tried to kill a kernel that is running by using cudaDeviceReset, cuCtxDestroy, and cuDevicePrimaryCtxReset. It seems that all of them wait for the kernel to finish and then
they probably release GPU resources. Is there any other solution? Is there an explanation?

Thank you in advance Manos.