Memory on GPU not freed after the program is killed.

I have a problem that the memory held by a process is not released after killing the process.
Here is the nvidia-smi status:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.30                 Driver Version: 390.30                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 00000000:02:00.0 Off |                  N/A |
| 23%   37C    P8    17W / 250W |    602MiB / 12196MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    Off  | 00000000:03:00.0 Off |                  N/A |
| 23%   35C    P8    10W / 250W |    536MiB / 12196MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    Off  | 00000000:82:00.0 Off |                  N/A |
| 23%   35C    P8    17W / 250W |    536MiB / 12196MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   3  TITAN X (Pascal)    Off  | 00000000:83:00.0 Off |                  N/A |
| 23%   32C    P8     8W / 250W |    600MiB / 12196MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     19453      C   py3.5                                        561MiB |
+-----------------------------------------------------------------------------+

The cards 2 & 3 have some memory held by an unknown process. Could someone help me with releasing the memory and handling scenarios like this?