RTX 5090 on Ubuntu 24.04: Xid 79 "GPU has fallen off the bus" after months of uptime, required power cycle

Graphics / Linux Linux
,
1

I have attached the nvidia-bug-report log captured after a failure.
I am seeing repeated GPU failures on a Linux workstation where the GPU falls off the PCIe bus during CUDA workloads.

System:

  • Machine: HP OMEN by HP 45L Gaming Desktop GT22-3xxx
  • Baseboard: HP 8D2C
  • BIOS: F.13, release date 09/10/2025
  • OS: Ubuntu 24.04.4 LTS
  • Kernel: 6.8.0-110-generic
  • GPU: NVIDIA GeForce RTX 5090 (GB202-A)
  • GPU UUID: GPU-dc4edb7d-05af-273a-4343-bd26a089db16
  • VBIOS: 98.02.79.00.09
  • NVIDIA driver: 580.126.09
  • CUDA version reported by nvidia-smi: 13.0

Problem:
During sustained CUDA workloads, the GPU becomes unavailable. After failure, nvidia-smi reports:

Unable to determine the device handle for GPU0: 0000:02:00.0: Unknown Error
No devices were found

The attached nvidia-bug-report log shows repeated Xid 79 events where the GPU falls off the bus. This has happened multiple times:

  • 2026-07-19: Xid 79, GPU has fallen off the bus
  • 2026-07-21: Xid 79, GPU has fallen off the bus
  • 2026-07-22: Xid 79, GPU has fallen off the bus

For the 2026-07-22 failure, the log shows a PCIe physical-layer AER error shortly before the Xid 79:

pcieport 0000:00:06.0: AER: Multiple Correctable error message received from 0000:00:06.0
pcieport 0000:00:06.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
pcieport 0000:00:06.0: [ 0] RxErr
NVRM: Xid (PCI:0000:02:00): 79, GPU has fallen off the bus.
NVRM: Xid (PCI:0000:02:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)

After the failure, the GPU PCI config space appears unreadable in the bug report, with “Unknown header type 7f” and all ff bytes.

The system is in a warm room, so overheating is possible, but I do not see explicit overtemperature or thermal shutdown messages in the logs. I would like to know whether this looks like a known issue with
RTX 5090 / driver 580.126.09 / Linux 6.8, or whether it should be treated primarily as PCIe/power/hardware instability.

Questions:

  1. Are there known issues with RTX 5090 on Linux driver 580.126.09 that can produce Xid 79 / Xid 154 under CUDA load?
  2. Do you recommend a different driver version for this GPU/kernel combination?
  3. Are there recommended kernel parameters, GSP firmware settings, PCIe ASPM settings, or BIOS PCIe generation settings to test?
  4. Does the PCIe AER RxErr immediately before Xid 79 suggest physical PCIe link instability rather than a software/driver issue?
2

that’s a legacy driver for Pascal and Maxwell: why would you use it for a Blackwell? I’d highly recommend upgrading to v595 or v610.

That’s really ancient… I’d recommend some LTS that was released after the Blackwell, for example 6.18.

I have never seen any such messages: GPUs just fall off the bus to my knowledge. You can try to watch thermals yourself, but in case you haven’t heard the news, the fact that the lying NV drivers report normal thermals, means absolutely nothing.

More generally, in the words of an NV eng:

So apart from thermals, make sure you have the latest BIOS/UEFI firmware for your mobo and that your PSU delivers sufficient amount of juice.

At this point I think it’s safe to say that 5090s and pro6000s do it on a semi-regular basis (regardless of the driver version). Just search this forum. This may be related to the hidden thermal issue, but not enough data yet to prove this.

Definitely, as stated at the beginning.

One of my business partners is experimenting with lowering max wattage to his pro6000 cards, but in such case you may as well go with AMD…

NV kernel modules are notorious for throwing a tantrum when a GPU falls off the bus and they usually destabilize the whole PCI kernel subsystem to the point that a reboot is the only option, so I wouldn’t be concerned about the AER (for example on my laptop it causes WiFi and NVMes to stop responding).