Inconsistent GPU utilization returned by nvidia-smi

Hello fellow Nvidia users,

We use Tesla T4 running on Ubuntu for video transcoding. In order to avoid overloading the GPUs we monitor the encode and decode utilizations. We’ve noticed that the readings returned by ‘nvidia-smi dmon’ and ‘nvidia-smi stats’ are inconsistent. The latter provides readings that are approx half of the former. See below where one returns 6% and the other 13-14%.

Why this inconsistency?

Note: We also use Tesla M60 and the numbers there are concistent across the 2 commands.

$ nvidia-smi stats -d encUtil -c 1
0, encUtil , 1593541909318963, 6
0, encUtil , 1593541909519612, 6
0, encUtil , 1593541909720366, 6
0, encUtil , 1593541909921131, 6
0, encUtil , 1593541910121965, 6

$ nvidia-smi dmon
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0    27    47     -     3     1    14     2  5000   585
    0    27    47     -     3     1    14     2  5000   585
    0    28    46     -     3     1    13     2  5000   585

$ nvidia-smi
Tue Jun 30 13:32:24 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64       Driver Version: 430.64       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   46C    P0    27W /  70W |    216MiB / 15109MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2313      C   ...ocal/WowzaStreamingEngine/java/bin/java   205MiB |
+-----------------------------------------------------------------------------+