I have a problem that the memory held by a process is not released after killing the process.
Here is the nvidia-smi status:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.30 Driver Version: 390.30 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN X (Pascal) Off | 00000000:02:00.0 Off | N/A |
| 23% 37C P8 17W / 250W | 602MiB / 12196MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 1 TITAN X (Pascal) Off | 00000000:03:00.0 Off | N/A |
| 23% 35C P8 10W / 250W | 536MiB / 12196MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 2 TITAN X (Pascal) Off | 00000000:82:00.0 Off | N/A |
| 23% 35C P8 17W / 250W | 536MiB / 12196MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
| 3 TITAN X (Pascal) Off | 00000000:83:00.0 Off | N/A |
| 23% 32C P8 8W / 250W | 600MiB / 12196MiB | 0% E. Process |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 19453 C py3.5 561MiB |
+-----------------------------------------------------------------------------+
The cards 2 & 3 have some memory held by an unknown process. Could someone help me with releasing the memory and handling scenarios like this?