NVIDIA-SMI 0 utilization while training

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

while training, NVIDIA SMI shows 0 utilization, I have noticed on both of DGX machines
any bug?

grep VERSION /etc/dgx-release
DGX_SWBUILD_VERSION="3.1.4"
DGX_OTA_VERSION="3.1.7"
DGX_OTA_VERSION="3.1.8"

sudo nvidia-smi
Wed Nov 13 04:47:31 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
| N/A   34C    P0    57W / 300W |   2101MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   35C    P0    57W / 300W |   1712MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
| N/A   35C    P0    58W / 300W |   1718MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:0B:00.0 Off |                    0 |
| N/A   31C    P0    57W / 300W |   1718MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 00000000:85:00.0 Off |                    0 |
| N/A   32C    P0    55W / 300W |   1718MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 00000000:86:00.0 Off |                    0 |
| N/A   32C    P0    55W / 300W |   1538MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 00000000:89:00.0 Off |                    0 |
| N/A   36C    P0    56W / 300W |   1538MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
| N/A   32C    P0    57W / 300W |   1538MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
2

Hi RaviKiranK,
I ran training with one gpu and not reproduced the 0 utilization.
Does it only happen in the DGX?

3

Hi Morgan,

I tried on 2 different DGX machines, the same results.

smi works with other frameworks

nvidia-smi
Thu Nov 14 03:31:56 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
| N/A   36C    P0    58W / 300W |   4784MiB / 32480MiB |     12%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   39C    P0    56W / 300W |   4010MiB / 32480MiB |     12%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
| N/A   39C    P0    58W / 300W |   4010MiB / 32480MiB |     12%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:0B:00.0 Off |                    0 |
| N/A   34C    P0    54W / 300W |   4010MiB / 32480MiB |     19%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 00000000:85:00.0 Off |                    0 |
| N/A   37C    P0    67W / 300W |   4004MiB / 32480MiB |     24%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 00000000:86:00.0 Off |                    0 |
| N/A   33C    P0    42W / 300W |     11MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 00000000:89:00.0 Off |                    0 |
| N/A   31C    P0    42W / 300W |     11MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
| N/A   29C    P0    42W / 300W |     11MiB / 32480MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     46646      C   /usr/local/bin/caffe                        3993MiB |
|    0     48121      C   /usr/bin/python                              383MiB |
|    0     54134      C   python                                       383MiB |
|    1     46646      C   /usr/local/bin/caffe                        3987MiB |
|    2     46646      C   /usr/local/bin/caffe                        3987MiB |
|    3     46646      C   /usr/local/bin/caffe                        3987MiB |
|    4     46646      C   /usr/local/bin/caffe                        3981MiB |
+-----------------------------------------------------------------------------+