RTX PRO 4000 Blackwell falling off the PCIe bus
I’ve been experiencing recurring hard system locks due to the RTX PRO 4000 Blackwell falling off the PCIe bus. This happens mostly with the system idle but sometimes during use as well. The system becomes completely unresponsive with the following symptoms:
Monitor goes completely blank with no error displayed
Full-screen virtual terminal (TTY3) not available
Keyboard becomes unresponsive (no response from any key)
Front panel power button does nothing
When SSHing into my workstation I can see that the GPU was disconnected. The command nvidia-smi returns an error indicating that the card was disconnected and cannot be found. The only way to recover is a full physical power cycle (unplug PSU).
System Configuration:
Chassis Model: Precision Tower 7910
OS: Ubuntu 22.04.5 LTS
Processor 1: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
Processor 2: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
Memory Total Installed Capacity: 262144 MB
GPU 1: NVIDIA RTX PRO 4000 Blackwell (10de:2c34)
GPU 2: NVIDIA Quadro M4000 (GM204GL)
Driver version: Driver Version: 580.178.04, CUDA Version: 13.0
As the RTX requires open nvidia drivers and the Quadro closed ones, I’m only running the RTX card and passingthrough the Quadro to QEMU/KVM. As a result, when the RTX disconnects the entire system goes down as there is no video output.
I tried different NVIDIA drivers (565, 570, 575, 595, etc.), to no avail. I’m running the card with very conservative settings and extremely low load, and yet the GPU crashes with no clear rhyme or reason. Here is the error message I get from “grep -i -e temp -e therm -e nvidia /var/log/syslog*”:
[388707.070964] nvidia-modeset: ERROR: GPU:0: Error while waiting for GPU progress: 0x0000ca7d:0 2:0:4048:4040
This seesm to shows that the GPU has fallen off the PCIe bus. While this is usually caused by power supply delivery, this is not an issue with my workstation, as it consumes around 400W and is powered by a 1200W PSU. I checked the power cables, which are not daisy-chained, and the connections are firm and stable.
I would appreciate any guidance or a firmware/driver fix for this issue.