I am trying to understand the following metrics for MIG instances:
- DCGM_FI_PROF_SM_ACTIVE 1002
- DCGM_FI_PROF_SM_OCCUPANCY 1003
nvidia-smi
Wed Feb 14 08:49:09 2024
±--------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------±---------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
±----------------------------------------±---------------------±---------------------+
| 3 NVIDIA A100-SXM4-80GB On | 00000000: Off | On |
| N/A 33C P0 85W / 400W | N/A | N/A Default |
| | | Enabled |
±----------------------------------------±---------------------±---------------------+
±--------------------------------------------------------------------------------------+
| MIG devices: |
±-----------------±-------------------------------±----------±----------------------+
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+================================+===========+=======================|
| 3 1 0 0 | 37MiB / 40192MiB | 42 0 | 3 0 2 0 0 |
| | 0MiB / 65535MiB | | |
±-----------------±-------------------------------±----------±----------------------+
| 3 2 0 1 | 49MiB / 40192MiB | 56 0 | 4 0 2 0 0 |
| | 0MiB / 65535MiB | | |
±-----------------±-------------------------------±----------±----------------------+
±--------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
±--------------------------------------------------------------------------------------+
In my experiment I am running the following workload:
dcgmproftester12 --no-dcgm-validation -i 3 -t 1004 -d 120
The GPU 3 is configured with two MIG instances.
To collect the metrics, I am running the following command:
dcgmi dmon -e 1002,1003 -g 18
#Entity SMACT SMOCC
ID
GPU-I 21 1.294 0.162
GPU-I 22 0.977 0.122
What I don’t understand is the metric DCGM_FI_PROF_SM_ACTIVE / SMACT / 1002.
Which is the ratio of cycles that the SM is active divided by the total number of cycles, so a value of 1 means 100% of utilization.
However, for MIG, dcgmi dmon is showing ~1.3, which is 130% of utilization.
So, what does it means?
So, is it a bug or there is a different interpretation for this metric?