Hi,
nvidia-smi shows 0MB as GPU memory utilization for all processes running in docker containers:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.28.03 Driver Version: 560.28.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3060 Off | 00000000:01:00.0 On | N/A |
| 53% 46C P0 27W / 170W | 2623MiB / 12288MiB | 4% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1991 G /usr/lib/xorg/Xorg 292MiB |
| 0 N/A N/A 2140 G /usr/bin/gnome-shell 212MiB |
| 0 N/A N/A 4354 G ...seed-version=20240905-050113.669000 272MiB |
| 0 N/A N/A 4964 G ...yOnDemand --variations-seed-version 52MiB |
| 0 N/A N/A 9484 C python3 0MiB |
| 0 N/A N/A 9491 C python3 0MiB |
| 0 N/A N/A 9497 C python3 0MiB |
| 0 N/A N/A 9525 C python3 0MiB |
+-----------------------------------------------------------------------------------------+
Of course, these docker processes use GPU and are therefore listed by nvidia-smi. Note that total memory utilizations 2623MiB seems to show a correct (or at least probable) value.
System details:
$ uname -a
Linux bluewaters 6.1.0-1028-oem #28-Ubuntu SMP PREEMPT_DYNAMIC Wed Dec 13 15:38:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Looks like a bug. How can I fix it?