RTX Pro 6000 Blackwell - Wrong temperature thresholds causes SW Power Cap, GPU throttled to 510MHz (1/6 performance)

Gaming and Visualization Technologies General Topics & Other SDKs
1

GPU: NVIDIA RTX Pro 6000 Blackwell
Device ID: 10de:2bb1 (rev a1)
VBIOS: 98.02.52.00.02
Driver: 570.211.01-server-open
OS: Ubuntu 22.04
Subsystem: NVIDIA Corporation Device 204b

Critical Issue:
Temperature thresholds are completely wrong:
GPU Shutdown T.Limit Temp : -5 C
GPU Slowdown T.Limit Temp : -2 C
GPU Max Operating T.Limit : 0 C

Because of these wrong thresholds, the driver believes the GPU
is always overheating, even at idle (actual temp: 32~38°C).
This permanently activates SW Power Cap, and causes abnormal
power readings far exceeding the rated limit:

Power Draw (idle) : ~1200W
Power Draw (loaded) : ~1219W
Current Power Limit : 600W
pviol : 100% (always)

GPU clock is throttled to ~510MHz instead of 3090MHz.
Actual GPU performance is reduced to about 1/6.
GPU is completely unusable for production workloads.

nvidia-smi dmon output during full workload (SM=100%):

gpu pwr gtemp sm mclk pclk pviol

 0   1218     38   100  13365    510    100

Attempted workarounds (all failed):

  • Reinstalled drivers (570, 580, 590, 570-server-open)
  • sudo nvidia-smi --lock-gpu-clocks=2617,2617
  • sudo nvidia-smi -pl 600
  • Reseated GPU physically

Root cause suspected: VBIOS 98.02.52.00.02 has incorrect
temperature threshold data, or driver misreads the thresholds.

Is VBIOS 98.02.55.00.00 available for Device ID 10de:2bb1?
Please advise urgently. This is a production system.

2

Welcome @user65806 to the NVIDIA developer forums.

Please check out Understanding Optimal GPU Temperature and Default GPU Fan Curve (NVIDIA RTX 6000 Ada)
for some detailed explanations on how to read the temperature values from nvidia-smi

And to help you better it would be good if you could share more complete output from nvidia-smi under load.

3

Hi, has this been solved in some way?
The VBIOS for your GPU would need to be provided by the distribution partner we sold the GPU to, so most likely like PNY or TDSynnex…
Though I doubt a VBIOS update would fix this, since the temp readings are NOT a bug/mistake; as outlined in the pointer Markus gave…

I’m still interested in the performance/clock degradation you described, it sounds similar to this posting:

Any chance, you could send me a full nvidia-smi log of the system, more logs and tests maybe..? many thanks
-Frank

4

I have encountered the same issue.

At 100% CUDA load, the card reports 600 W, asserts clocks_event_reasons.sw_power_cap, and throttles to 652–675 MHz at 35–37 °C.
Thermal and hardware slowdown flags remain inactive; no Xid, ECC, or PCIe
errors occur. Suspected onboard power-sensor/PMIC telemetry failure.

Also, I noticed that the idle power is surprisingly high 200-300 W without any workloads, which is also mentioned in other threads.