This is a Threadripper PRO workstation with two Blackwell cards in it, and since 31 August one of them has abruptly disappeared from the PCIe bus seven times. The messages come in the same order every time. The PCIe root port reports a DPC containment event, which means the link went down. A moment later the NVIDIA driver reports Xid 79, GPU has fallen off the bus, and then Xid 154, which sets the recovery action for the node to Node Reboot Required. Service restarts and driver reloads never bring the card back. Only a full reboot does.
A few things make this hard to pin down. The two cards are different products, an RTX PRO 6000 Blackwell Workstation Edition and a GeForce RTX 5090, and both have failed with the same sequence of messages. And in every event with telemetry, it is the card that was working that leaves the bus, never the idle one.
There is no warning on the link side. This host samples 16 PCIe link-layer counters per GPU once a second, and the last sample before the drop, taken 0.6 seconds before the event, still shows Gen5 x16 with every one of those counters at zero. With a one-second interval the final 0.6 seconds is not covered by any sample, so what I describe is the 30 minutes of samples that are covered. On the card side there are small changes in the last second, and those are listed further down. What the link layer never shows is a warning: it goes from clean to gone. The other direction does not point at the victim either, at least not in the last event, where the card that survived was running hotter and drawing more, 599 W at 80-81 C, against 563 W at 71-72 C for the card that died.
The machine is an ASUS Pro WS WRX90E-SAGE SE, Rev 1.xx, BIOS 1317, with an AMD Ryzen Threadripper PRO 9975WX, running Ubuntu 26.04.1 on kernel 7.0.0-38-generic. The driver is 595.99.02, open kernel module, CUDA 13.2. The RTX PRO 6000 Blackwell Workstation Edition sits at 0000:11:00.0 behind root port 0000:10:01.1, VBIOS 98.02.81.00.07. The GeForce RTX 5090 sits at 0000:f1:00.0 behind root port 0000:f0:01.1, VBIOS 98.02.2E.40.34. Both cards are set to a 600 W power limit on a Gen5 x16 link, and each card runs one vLLM instance for continuous LLM inference.
This is what the most recent event looks like in the kernel log:
17:26:47 pcieport 0000:f0:01.1: DPC: containment event, status:0x1f01: unmasked uncorrectable error detected
17:26:47 pcieport 0000:f0:01.1: PCIe Bus Error: severity=Uncorrectable (Fatal), type=Transaction Layer, (Receiver ID)
17:26:47 pcieport 0000:f0:01.1: device [1022:153e] error status/mask=00000020/00000000
17:26:47 pcieport 0000:f0:01.1: [ 5] SDES (First)
17:26:47 nvidia 0000:f1:00.0: AER: can't recover (no error_detected callback)
17:26:47 snd_hda_intel 0000:f1:00.1: AER: can't recover (no error_detected callback)
17:26:48 NVRM: GPU at PCI:0000:f1:00: GPU-<redacted>
17:26:48 NVRM: Xid (PCI:0000:f1:00): 79, GPU has fallen off the bus.
17:26:48 NVRM: GPU 0000:f1:00.0: GPU serial number is 0.
17:26:48 NVRM: Xid (PCI:0000:f1:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)
17:26:48 NVRM: Xid (PCI:0000:11:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)
17:26:48 pcieport 0000:f0:01.1: broken device, retraining non-functional downstream link at 2.5GT/s
17:26:49 pcieport 0000:f0:01.1: AER: device recovery failed
In plain terms, the root port sees the link go down. The only bit set in the error status is 0x20, which is SDES, bit 5, meaning the link went down unexpectedly rather than a data error being detected, so this signature can appear with every per-lane error counter at zero. In one of the seven events the status also had the completion-timeout bit set alongside SDES; the other six had SDES on its own. The whole endpoint disappears at once, both the GPU function f1:00.0 and its audio function f1:00.1. The driver gives up and marks the node as needing a reboot, and when the root port tries to retrain the link it drops to Gen1 speed and fails there too.
The seven events, with the power reading from the last one-second sample before each one:
- 2026-08-31 13:33:18, RTX PRO 6000, root port 10:01.1 (no telemetry was being collected yet at that point)
- 2026-09-14 10:58:11, RTX PRO 6000, root port 10:01.1, 525 W
- 2026-09-14 13:59:16, RTX PRO 6000, root port 10:01.1, 549 W
- 2026-09-14 16:02:31, RTX PRO 6000, root port 10:01.1, 471 W
- 2026-09-14 16:12:30, RTX PRO 6000, root port 10:01.1, 565 W
- 2026-09-30 21:35:14, RTX 5090, root port f0:01.1, 531 W
- 2026-10-14 17:26:48, RTX 5090, root port f0:01.1, 563 W
The last of these cost 28 minutes and 45 seconds of downtime, from 17:26:47 until the machine came back at 17:55:32. Once the node is rebooted the card comes back at Gen5 x16 and runs normally until the next event.
For the first event there is no telemetry. The only other signal around it is the BMC slot sensor for that card (PCIE04), which read 90-91 C. Following that alarm path in the BMC firmware, it contains no power-off or link-disable action, so it records a reading rather than causing the cut-off. The BMC log timestamps are not reliable enough to order that entry against the DPC line.
The sampler records, per GPU and once a second: power, temperature, PCIe link generation and width, and the NVML PCIe field counters for replay count, replay rollover, L0 to Recovery, lane errors, correctable errors, correctable error rate, NAKs received and sent, receiver error, bad TLP, bad DLLP, LCRC error, unsupported request and fatal errors.
For the most recent event, here is the victim next to the card that survived, over the same 30 minutes.
The card that died, the RTX 5090, held Gen5 x16 the whole time with zero L0 to Recovery transitions, and every error counter above was zero at every sample, right up to the last one 0.6 seconds before the event. Its power stayed between 542 and 569 W and its temperature between 71 and 72 C. At that moment vLLM was serving 10 running and 22 queued requests with the KV cache at 93 to 95 percent, generating around 1,572 to 1,618 tokens per second. Once the event happened its telemetry stopped updating: power and temperature froze at their last values and link width read as “None”.
The card that survived, the RTX PRO 6000, did 18 L0 to Recovery transitions in the same 30 minutes, cycling Gen5 to Gen2 to Gen1 and back as it went briefly idle, and every error counter stayed at zero. Its power cycled between 10 W and 600 W and its temperature ran at 80-81 C, and it kept serving requests throughout.
Across all six events that have telemetry, the numbers look like this. On each line, the victim first, then the survivor, as L0 to Recovery transitions, link generation changes, and power at the last sample:
- 09-14 10:58, victim the PRO 6000: 15 transitions, 15 generation changes, 525 W. The 5090 had 53, 50, 25 W.
- 09-14 13:59, victim the PRO 6000: 10, 10, 549 W. The 5090 had 15, 15, 462 W.
- 09-14 16:02, victim the PRO 6000: 16, 16, 471 W. The 5090 had 5, 5, 23 W.
- 09-14 16:12, victim the PRO 6000: 3, 3, 565 W. The 5090 had 5, 5, 24 W.
- 09-30 21:35, victim the 5090: 3, 3, 531 W. The 6000 had 30, 30, 7 W.
- 10-14 17:26, victim the 5090: 0, 0, 563 W. The 6000 had 18, 18, 599 W.
Two things stand out in that list. In every event with telemetry the victim’s utilisation at the last sample was 97 to 100 percent and its power was 471 W or more, while the survivor was idle at 7 to 25 W in four of the six. And link retraining is routine here rather than a fault in itself: the surviving card in the last event did 18 down and up transitions in 30 minutes with no errors at all, and in five of the six events the card that died had fewer link transitions than the card that lived. “The link retrains and fails” does not describe what happens on this machine.
The card side does move a little. Taking the five events before the last one, where the whole window was analysed: the temperature rises in the final second in some events and not others (+4, 0, +7, 0 and -1 C across them), the PRO 6000 events reach 89 to 90 C somewhere in that window while the 5090 events stay at or below 81 C, and the NVML clock event reason field is nonzero at the end, usually the software power cap bit and in one event a hardware slowdown bit as well. None of that is consistent enough to call a trigger, and the drop does not happen at peak power: in all five of those events the reading at the moment of death, 471 to 566 W, is below the peak of the last five minutes in the same window, which is 590 to 606 W.
What the data points away from, and why:
- A fixed power or temperature threshold. The four PRO 6000 events on 14 September died at 86 to 90 C and the two 5090 events at 79 C and 72 C, and in the five earlier events the power at the moment of death was below the peak of the final five minutes.
- Link degradation before the event. All 16 PCIe counters on the victim were zero in all six events, including the last sample 0.6 seconds before the death.
- Link retraining by itself. Both cards retrain constantly: the survivor did 18 down and up transitions in 30 minutes with zero errors, and in five of the six events the victim retrained less than the card that survived.
- A whole-host problem. In the same capture the other card’s path stayed healthy, and there are no host MCE, GHES, APEI, EDAC, pstore or vmcore records for that window.
- A single bad card. Two different GPU models, workstation and consumer, fail with the same sequence on this machine.
That list comes from this machine’s own measurements. For context, the signature is not unique to this hardware either: one other user reports it on unrelated platforms, listing WRX80, WRX90, EPYC, X670E, AM4 and Z490, and reporting that driver versions 570, 575, 580 and 595 all still show it.
What I still cannot see:
- No bug report was captured in any of these events. The reboots were manual and there was no automatic capture step; adding one is the first thing to fix on this side.
- A secondary bus reset or an endpoint rescan has not been tried; the attempts so far were service restarts and driver reloads.
- The telemetry is one sample per second. Anything that happens faster than that is invisible in this data set.
A few questions for NVIDIA:
- Does this match a bug you are already tracking? I have seen other reports of this signature, including two threads in this forum (RTX 5090 (GB202) + Ryzen 9 9950X3D: Xid 79 at idle, reproduced across 2 driver branches, ASPM confirmed disabled at register level, zero AER errors and Deterministic Xid 120 GSP kernel panic on RTX PRO 6000 Blackwell under sustained cuBLAS workload (595.84)) and RTX 5080 (GB203): Random Xid 79 with zero precursor — any load level, instant atomic GPU death, open kernel module 595.71.05 · Issue #1151 · NVIDIA/open-gpu-kernel-modules · GitHub, and I would like to know whether this is being tracked as a Linux driver or GSP issue, or as a hardware issue on the card side.
- Is there a supported way to recover without a full node reboot, such as a secondary bus reset, nvidia-smi --gpu-reset, or unbind and rebind, or is a reboot the only path?
- Your 615.71.09 release notes say: “NVML/nvidia-smi was calling into a library to get some extended PCI info fields for each GPU. Under unique conditions, this can result in GPU fallen off bus errors in low frequency. The fix is to remove the calls to that library in high frequency paths like nvidia-smi -q. The library call will be replaced with a standard RM control call.” This host runs 595.99.02, and the sampler polls NVML_FI_DEV_PCIE_* field counters, 16 per GPU, once a second. Is that a possible contributing factor for this signature, or is that note about a different failure mode? I am reluctant to turn off the only telemetry that captures these events without knowing.