Hi @abchauhan, here’s an update in case it helps with bug #6426268 or anyone else hitting this.
Recap
Card: RTX PRO 6000 Blackwell Workstation Edition, Dell-channel board 900-5G144-0100-000, VBIOS 98.02.52.00.03.
Driver: 610.57.04 (open kernel modules). Ubuntu, used for LLM inference.
What happened: the GSP firmware hung 7 times between 17 Aug and 5 Sep. The heartbeat timed out, then Xid 175 (GSP_RM_CONTROL RPC timeout), then Xid 154 (PF FLR). Only a full power cycle recovered it.
It’s the card, not the slot. After we swapped the two cards between slots, the fault moved with this board. Our second card (VBIOS 98.02.81.00.07) has never wedged.
GSP program counter: 4 of the 5 captured hangs have riscvPc 01aa13aa. The last three all died on control command 0x2080a61d.
The pattern that led to the change
All 7 hangs started while the card was idle or nearly idle, sitting in P8 at about 180 MHz. None happened under sustained load. The first one came 33 minutes after a long batch job finished. We had already ruled out several causes:
Temperature: the card was at 37–46 °C at every onset.
PCIe: AER was clean.
Power cap: it wedged at 300 W and at 400 W.
VRAM pressure, and churn from monitoring tools that repeatedly open driver (RM) connections: we cut that churn to almost nothing and the wedges continued.
That left the idle P-state / clock-change path as the obvious suspect.
The change
On 6 Sep at 07:53 UTC we locked the graphics clock on the affected card only:
We apply it from a systemd unit at boot because -lgc doesn’t survive a reboot. The memory clock is not locked. The card now idles in P1 at about 2295 MHz and draws about 80–87 W at idle, up from about 20 W. Under load it still throttles below the lock to stay under the 400 W cap, so in practice the lock only acts as a floor while the card is idle. It keeps the card out of the low P-states.
Result
Before the lock With the lock
Period 17 Aug – 5 Sep (~19 days) 6 Sep – 23 Sep (17 days, ongoing)
GSP hangs 7 0
Longest clean run 5.5 days 17 days so far
Xid / NVRM errors many none
At the old rate, which averaged under 3 days between hangs, a 17-day clean run would be very unlikely to happen by chance.
Caveats, to be fair
This wasn’t a clean single-variable test:
On the same morning we moved fan control to one persistent NVML session, replacing periodic nvidia-settings calls.
On 13 Sep the workload changed from vLLM to SGLang.
On 17 Sep the host moved to a new chassis, with a planned shutdown and a new slot.
A trial we’d queued to turn on ECC was cancelled and never took effect.
So the clock lock is our leading explanation, but I can’t separate it completely from the fan-control change. We’re leaving everything as it is and will report back if it wedges again.
Questions
Does “hangs only at idle, stopped by keeping the card out of low P-states” fit what you know about #6426268? Is there a known GSP issue with P-state or DVFS changes at idle on this SKU?
Is a firmware or driver fix expected, and will it come in the driver’s GSP firmware or need a VBIOS update?
Is there a newer VBIOS for this Dell-channel board (900-5G144-0100-000, currently 98.02.52.00.03)? The public Workstation updater won’t accept it.
For anyone else hitting this: locking the graphics clock with nvidia-smi -lgc may be worth trying as a workaround. It costs about 60 W extra at idle, and it doesn’t affect performance under load once the power cap is limiting the card anyway.
Thanks for your help.