RTX PRO 4000 Blackwell falling off the PCIe bus

RTX PRO 4000 Blackwell falling off the PCIe bus

I’ve been experiencing recurring hard system locks due to the RTX PRO 4000 Blackwell falling off the PCIe bus. This happens mostly with the system idle but sometimes during use as well. The system becomes completely unresponsive with the following symptoms:

Monitor goes completely blank with no error displayed
Full-screen virtual terminal (TTY3) not available
Keyboard becomes unresponsive (no response from any key)
Front panel power button does nothing

When SSHing into my workstation I can see that the GPU was disconnected. The command nvidia-smi returns an error indicating that the card was disconnected and cannot be found. The only way to recover is a full physical power cycle (unplug PSU).

System Configuration:

Chassis Model: Precision Tower 7910
OS: Ubuntu 22.04.5 LTS
Processor 1: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
Processor 2: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
Memory Total Installed Capacity: 262144 MB
GPU 1: NVIDIA RTX PRO 4000 Blackwell (10de:2c34)
GPU 2: NVIDIA Quadro M4000 (GM204GL)
Driver version: Driver Version: 580.178.04, CUDA Version: 13.0

As the RTX requires open nvidia drivers and the Quadro closed ones, I’m only running the RTX card and passingthrough the Quadro to QEMU/KVM. As a result, when the RTX disconnects the entire system goes down as there is no video output.

I tried different NVIDIA drivers (565, 570, 575, 595, etc.), to no avail. I’m running the card with very conservative settings and extremely low load, and yet the GPU crashes with no clear rhyme or reason. Here is the error message I get from “grep -i -e temp -e therm -e nvidia /var/log/syslog*”:

[388707.070964] nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040

This seesm to shows that the GPU has fallen off the PCIe bus. While this is usually caused by power supply delivery, this is not an issue with my workstation, as it consumes around 400W and is powered by a 1200W PSU. I checked the power cables, which are not daisy-chained, and the connections are firm and stable.

I would appreciate any guidance or a firmware/driver fix for this issue.

It’s a known problem with higher-end Blackwell chips, see:

A workaround is to limit max clock speed and/or wattage as described in the links above.

Thanks, Morgwai. I read a few of these threads which informed my reporting here. What is different in my case is that the issue is happening when the GPU is idle with no load whatsoever and reporting only 14W consumption (through nvidia-smi). This seems different from the other cases when Blackwell GPUs have fallen off the PCI bus.

EDIT: some users in the GitHub thread are reporting issues with the Blackwell 5080 or 5090 at idle, but there doesn’t seem to be a workaround or fix for this.

I remember reading somewhere that in case of falling when idle, capping the max clock also helped and it was speculated that sudden short clock boosts are to blame. IMO, just give it a try and see if it helps ;-)

I’ll give it a go. I’ve tried pretty much everything else so might as well try that. I’ll also swap the slots of the RTX with the Quadro, which is currently in the main PCI slot. Some users seem to have had better luck with the RTX in the primary slot (in my case both are PCIe x16 with independent processors handling them so I’m not sure this will make any difference).