RTX 5090 (open 595.71.05, Ubuntu 24.04): bursty small-kernel launches corrupt GPU front-end command decode in ~3 s (Xid 13 SKED/class-mismatch, Xid 32

Graphics / Linux Linux
1

I have a 100%-reproducible GPU front-end corruption on an RTX 5090 that triggers in ~3 seconds with a minimal pure-PyTorch script (inlined below and in the attached zip, ~50 lines, no convs, no frameworks beyond torch). I’ve spent several days eliminating host-side causes and I believe this is either a driver/GSP bug on Blackwell or a defective GPU front-end; I’d appreciate help discriminating the two.

This looks identical to the unresolved report in RTX 5090 Xid 13 illegal instruction encoding with driver 580-open on Ubuntu 24.04 — filing a new topic since my evidence set is large; I’ll cross-link there.

System

  • GPU: RTX 5090 32 GB (Blackwell, sm_120), PCIe x16 (Gen4 forced in BIOS, also fails at Gen5 auto)
  • Driver: 595.71.05 open kernel module (also reproduced on 580.159.03-open), CUDA 13.0
  • OS: Ubuntu 24.04.4, kernel 6.17.0-40-generic (HWE), X11/GNOME
  • Board: MSI PRO X870-P WIFI (AMD X870), BIOS 1.A82 (2026-03-19; also reproduced on 1.A70)
  • RAM: 2×16 GB DDR5 at JEDEC 4800 (EXPO off), memtest86+ overnight PASS
  • PyTorch 2.13.0+cu130

Symptom

Any workload with a bursty small-kernel launch cadence (e.g. YOLO inference/training, or the reproducer below) kills the CUDA context within seconds: torch.AcceleratorError: CUDA error: unspecified launch failure. Steady heavy compute (8k×8k matmul loop, 90 s+) runs fine. The kernel log shows front-end command-decode garbage, a different flavor each time:

  • Xid 13 with ~8 SKED sanity-check failures at once (SKEDCHECK02/03/05 local-memory sizes, 06_REGISTER_COUNT, 11_TOTAL_THREADS, 16_CTA_THREAD_DIMENSION_ZERO, 23_CONSTANT_BUFFER_SIZE, XX_MISC) — i.e. a garbage QMD/launch descriptor
  • Xid 13 “Graphics Exception: Class 0xffff Subchannel 0x0 Mismatch” (ESR 0x4041b0=0x3f00ffff / 0x3f20ffff, ESR 0x404000=0x80000002, channel 0x2, Class 0000cec0)
  • Xid 32 invalid/corrupted push buffer, Data 0xdeaddead
  • Xid 69 Class Errors on Xorg’s channel (see key observation below)

Worst case it escalates to a full system freeze: Xid 13 → 43 → 69 → 119 (GSP RPC timeout) → 154 (GPU Reset Required), RC watchdog “GPU is probably locked!” spam, hard reset required. Attempting nvidia-smi against the wedged GSP reliably triggers the Xid 175 → 154 cascade.

Separately, the card has now twice failed to initialize at all on a cold boot (zero load, blank monitor): Xid 79 “GPU has fallen off the bus” ~2 s after the nvidia module loads (pid=udev-worker, i.e. during driver attach), followed by GSP heartbeat timeout with all MMIO reads returning 0xffffffff and RmLogGpuCrash “failed to save GPU crash data”. PCI config space still enumerates (device visible in lspci, VBIOS readable) but Mem-/BusMaster- and all BARs disabled. A subsequent full power-off boot comes up normally. First occurrence 2026-07-16, second 2026-07-17 22:28. (Possibly related: thread 369440 reports Xid 79 on this same motherboard, attributed there to thermals — which can’t apply here: cold boot, zero load. Reseating the card and 12V-2x6 cable changed nothing; see below.)

Also note: after a compute crash, the wedged GSP state survives a warm reboot. On the next boot the driver logs “unexpected WPR2 already up, cannot proceed with booting GSP” / “GPU lost from the bus [NV_ERR_GPU_IS_LOST]” while polling GSP_INIT_DONE, and RmInitAdapter fails (0x62:0x40:2168) — no display until a full cold power-off (PSU switched off) clears it (observed 2026-07-17 23:49; cold power-off restored a clean boot, verified 2026-07-18).

Minimal reproducer (l1_burst_tiny.py, also in the zip)

Pure PyTorch: bursts of 400 tiny kernels (64×64 matmuls, elementwise, reductions, fp16 matmul, gather) with 30 ms idle gaps between bursts. Crashes in 2.6–3.1 s (~20k launches) on every unserialized run, across multiple boots, two driver versions, and two BIOS versions. 180 s target duration; it has never survived past ~4 s.

"""L1: bursty tiny-kernel launches, pure PyTorch. No convs, no ultralytics.

Mimics the launch pattern of YOLO inference (many small kernels, rapid
idle<->active transitions) with none of the software stack on top.
"""
import sys
import time

import torch

DURATION_S = 180
BURST_KERNELS = 400   # small launches per burst
IDLE_GAP_S = 0.03     # idle gap between bursts, forces P-state churn

assert torch.cuda.is_available()
dev = torch.device("cuda:0")
torch.manual_seed(0)

# Small tensors of varied shapes/dtypes so launch descriptors (QMDs) vary,
# like a real model's mixed kernel stream.
a = torch.randn(64, 64, device=dev)
b = torch.randn(64, 64, device=dev)
c = torch.randn(1, 256, device=dev)
h = torch.randn(128, 128, device=dev, dtype=torch.float16)
idx = torch.randint(0, 64, (32,), device=dev)

print(f"L1 burst-tiny: {BURST_KERNELS} kernels/burst, {IDLE_GAP_S}s gaps, "
      f"{DURATION_S}s total", flush=True)
start = time.time()
bursts = 0
launches = 0
try:
    while time.time() - start < DURATION_S:
        for i in range(BURST_KERNELS // 8):
            x = a @ b                    # small matmul
            x = torch.relu(x + b)        # elementwise x2
            y = x.sum(dim=0)             # reduction
            z = h @ h                    # fp16 matmul
            w = torch.sigmoid(c) * c     # elementwise x2
            v = x[idx]                   # gather
            launches += 8
        torch.cuda.synchronize()
        bursts += 1
        if bursts % 25 == 0:
            el = time.time() - start
            print(f"  t={el:5.1f}s bursts={bursts} launches~{launches}", flush=True)
        time.sleep(IDLE_GAP_S)
    torch.cuda.synchronize()
    print(f"PASS: {bursts} bursts, ~{launches} launches, no error", flush=True)
except Exception as e:
    el = time.time() - start
    print(f"CRASH at t={el:.1f}s after ~{launches} launches:", flush=True)
    print(f"  {type(e).__name__}: {e}", flush=True)
    sys.exit(1)

Key observation: fault is cross-channel / cross-process

With CUDA_LAUNCH_BLOCKING=1, the same reproducer passes the full 180 s (~1.82M launches, zero compute errors) — but during that run the fault fired anyway, on Xorg’s channel instead, at t≈6 s (exactly the usual crash window), and again ~4 s after the run ended:

NVRM: Xid (PCI:0000:01:00): 69, pid=2396, name=Xorg, Class Error: channel 0x00000005, Class 0000ce97, Offset 00001618, Data 00000007, ErrorCode 0000000d
NVRM: Xid (PCI:0000:01:00): 69, pid=2396, name=Xorg, Class Error: channel 0x00000005, Class 0000ce97, Offset 00002384, Data ffffffff, ErrorCode 0000000c

So serializing one process’s launches protects that process, but whichever other channel is active takes the corruption instead. Combined with the rotating error flavors (garbage QMD / class mismatch / deaddead pushbuffer), this looks like corruption in the shared host-interface/front-end path (PBDMA/ESCHED/GSP), not in any one application’s command stream.

Ruled out (each verified, then reproducer re-run)

  • Driver version: 580.159.03-open and 595.71.05-open both crash identically (595 release notes mention a Blackwell Xid 13 TMA-descriptor fix — not this)
  • BIOS/AGESA: 1.A70 and 1.A82 both crash identically (3.1 s / ~19.6k launches on 1.A82)
  • PCIe generation: Gen5 auto and Gen4 forced both crash; PCIe replay counter and all AER counters remain 0 through every crash
  • System RAM: memtest86+ overnight PASS; EXPO never enabled (JEDEC 4800); no EDAC/MCE
  • Clocks/thermals/power: crashed in ~10 s with clocks locked at 2200 MHz — GPU at 76 W / 42 °C at the moment of crash. Also crashes at stock in P1 at ~90 W / 33 °C.
  • AMP/framework stack: amp=False doesn’t help; reproducer is pure torch (no ultralytics, no convs); steady heavy compute is stable
  • Card seating / power connector: after the second Xid 79 init failure I fully powered off and reseated the card and the 12V-2x6 cable; next boot was clean but the reproducer still crashed identically at t=2.6 s (~18.8k launches), same Xid 13 Class 0xffff Subchannel Mismatch (ESR 0x4041b0=0x3f00ffff, channel 0x2, Class 0000cec0)
  • One more datapoint from the attached bug report: the PCIe link never trains above Gen3 — GPU Link Info shows Max: 3 with Host Max: 4 (Gen4 forced) and Device Max: 5 — across both BIOS versions and unchanged by the card/cable reseat, while Replay/AER counters stay at 0. Reads like card-side PHY/host-interface trouble rather than platform signal integrity.

Cross-OS update: I ran the same pure-PyTorch bursty small-kernel reproducer on
Windows (610.47 WDDM / CUDA 13.3 — a stack that shares almost nothing with the
Linux open driver), and it crashed at t~4.5 s with cudaErrorLaunchFailure,
logging nvlddmkm Event 13 + 153 (GPU engine exception + reset) at the crash
instant. That’s the same front-end fault as Linux (Xid 13/32 within ~3 s), same
launch cadence, on an independent driver stack — so this isn’t a Linux
driver/GSP bug, it’s the card. Everything else was already ruled out on Linux
(driver 580/595, BIOS, PCIe gen 4/5, overnight memtest, and a locked 2200 MHz /
76 W / 42 °C run still crashed within seconds).

Questions

  1. Is this a known GSP/driver issue on Blackwell (beyond the 595 TMA fix)? Thread 349791 (RTX 5090, Xid 13, 580-open, Ubuntu 24.04) looks identical and unresolved.
  2. Are there GSP logs/flags I can capture that would discriminate a driver bug from a defective front-end (before I pursue an RMA)?
  3. The attached zip contains the reproducer, four run logs (stdout + gpumon CSV + kernel-log excerpts) spanning both BIOS versions and the reseat test, and nvidia-bug-report.log.gz captured minutes after the two Xorg Xid 69 hits above, same boot.

Serials are in the attached logs

rtx5090-xid13-evidence.zip (378.1 KB)

RTX 5090 - Xid 13 Illegal Instruction Encoding with nvidia-driver-580-open on Ubuntu 24.04