Hi,
we have different GraceHopper compute nodes; some are QCT systems (i.e. QuantaGrid S74G-2U) and some others are Supermicro (i.e. ARS-111GL-NHR) systems.
As far as I understood the GraceHopper chip in all the machines should be the same, but we measure different performance values, which are lower in the latter systems with respect to the former.
We probably found the issue, which is related to the fact that on the Supermicro machines apparently there is 700W power cap, while this is set to 900W on the QCT machines:
QCT:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 GH200 480GB On | 00000009:01:00.0 Off | 0 |
| N/A 52C P0 375W / 900W | 2382MiB / 97871MiB | 99% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
Supermicro:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GH200 480GB On | 00000009:01:00.0 Off | 0 |
| N/A 23C P0 64W / 700W | 0MiB / 97871MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
From this post I see that also on the Supermicro machine we should have a 900W maximum power:
And nvidia-smi reports a default Max Power of 900W:
$ nvidia-smi -q -d POWER
==============NVSMI LOG==============
Timestamp : Mon Jun 30 14:39:54 2025
Driver Version : 575.57.08
CUDA Version : 12.9Attached GPUs : 1
GPU 00000009:01:00.0
GPU Power Readings
Average Power Draw : 64.46 W
Instantaneous Power Draw : 64.34 W
Current Power Limit : 700.00 W
Requested Power Limit : 700.00 W
Default Power Limit : 900.00 W
Min Power Limit : 100.00 W
Max Power Limit : 900.00 W
Using nvidia-smi we can lower the power limit under 700W, but apparently we are not able to increase it to 900W:
# nvidia-smi -pl 600
Power limit for GPU 00000009:01:00.0 was set to 600.00 W from 700.00 W.
All done.
# nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GH200 480GB On | 00000009:01:00.0 Off | 0 |
| N/A 24C P0 64W / 600W | 0MiB / 97871MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
# nvidia-smi -pl 900
Power limit for GPU 00000009:01:00.0 was set to 900.00 W from 600.00 W.
All done.
# nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GH200 480GB On | 00000009:01:00.0 Off | 0 |
| N/A 24C P0 64W / 700W | 0MiB / 97871MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
Any ideas about what could be forcing a Power Cap of 700W on the Supermicro GraceHopper GPU?
Thanks,
Enrico