How To Recover From A Memory Leak

I have a few blocks of memory that were never freed before the kernel crashed.
i.e. Missing cudaFree(); or thrust:device_free();
How can I get this memory back without rebooting the machine?

Does the cudaRT give any/all memeory back when there is a crash? i.e. As part of a cleanup routine?
Is there a cuda command that will reset the memory alloc table?
Is there a command that will show the memory stat’s?


Normally all CUDA memory allocated by a process is freed when the process exits. If you are not seeing this, then there is a driver bug and I would suggest reporting it on the registered developer site.

“nvidia-smi -l” (that’s lowercase L if your font doesn’t have serifs) will show you a pretty useful table of memory usage, temperature, and fan speed. On GeForce GPUs it does not show GPU load or process usage, unfortunately.