I have seen other similar questions. here is an example. There are others. I don’t have any further suggestions to add other than what I have shared already. I doubt there is a precise, deterministic, guaranteed method to fix this observation in every imaginable case, other than a reboot. As you can see from that other thread, there may be other processes that need to be killed before the GPU will recover. Until the GPU is recovered, by reboot or some other method, I don’t know of a specific method to guarantee that other processes using that GPU will behave normally.
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| How to cleanly kill a CUDA application | 5 | 5026 | September 30, 2016 | |
| stuck CUDA program how to restart GPU when CUDA gets stuck | 0 | 1328 | August 10, 2010 | |
| any way to kill a gpu process ? | 1 | 6617 | July 1, 2009 | |
| Computation crash = stuck at 574mhz | 9 | 1289 | August 4, 2015 | |
| How to terminate a GPU program | 2 | 24521 | March 31, 2011 | |
| Is there any way to implement RR scheduling algorithm on the GPU? | 1 | 319 | August 16, 2019 | |
| Trouble killing CUDA processes? | 1 | 6076 | July 8, 2008 | |
| Terminate CUDA kernel which got stuck in an endless loop? Is that possible under linux? | 9 | 7573 | December 20, 2008 | |
| Kernel Interruption in Command Line Application | 1 | 7375 | July 15, 2011 | |
| Failure with independent devices on independent processes Try it yourself! | 19 | 3464 | March 10, 2011 |