Repeated system crash Ubuntu 24.04, Dual 5090 setup

Graphics / Linux Linux
1

Hello all,

I am reporting a critical system stability issue affecting dual NVIDIA GeForce RTX 5090 GPUs setup. This appears to be related to the known GSP firmware bug affecting Blackwell (RTX 50 series) GPUs. Please help to identify the issue.

SYSTEM INFORMATION

GPU: 2x NVIDIA GeForce RTX 5090
Driver: 580.126.09 (open kernel module)
OS: Ubuntu 24.04.3 LTS
Kernel: 6.17.0-22-generic
Motherboard: GIGABYTE TRX50 AI TOP(firmware F12g 03/31/2026)
CPU: AMD Ryzen Threadripper 7970X (32-core)
RAM: 256GB

PROBLEM DESCRIPTION

The system experiences HARD FREEZES requiring a physical power cycle to recover.
This occurs RANDOMLY, often during IDLE or low GPU load conditions.

Symptoms:

  • Complete system lockup (no SSH, no keyboard, no display)
  • Logs stop abruptly mid-write
  • No kernel panic or crash dump generated
  • Requires physical power button to recover

When logs are preserved (rare), the following errors appear:

[    9.072467] kernel: NVRM: GPU at PCI:0000:41:00: GPU-8f4499d4-d4fb-126f-65b5-d6319979c633
[    9.072471] kernel: NVRM: Xid (PCI:0000:41:00): 79, GPU has fallen off the bus.
[    9.072479] kernel: NVRM: kgspRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.
[    9.072679] kernel: NVRM: _kgspLogRpcSanityCheckFailure: GPU1 sanity check failed 0xf waiting for RPC response from GSP.
               Expected function 4097 (GSP_INIT_DONE) sequence 0 (0x0 0x0).
[    9.122798] kernel: NVRM: osInitNvMapping: *** Cannot attach gpu
  1. Is this a known GSP firmware issue affecting dual-RTX-5090 configurations?
  2. Is there a driver update (595.x or newer) that addresses this GSP timeout?
  3. Can you confirm if disabling GSP (nvidia.NVreg_EnableGpuFirmware=0) is safe for RTX 5090?
  4. Are there any kernel parameters or workarounds to prevent GSP timeouts?
  5. Is a driver fix planned for the 580/595 series?

This issue matches multiple confirmed bug reports:

  1. “RTX 5070 (10de:2f04) — Spontaneous GSP RPC timeout, GPU lost from bus on 580.126.18”
    RTX 5070 (10de:2f04) — Spontaneous GSP RPC timeout, GPU lost from bus on 580.126.18

  2. “[Bug] Hard System Lockup / GPU Lost From Bus under Load - RTX 5070”
    [Bug] Hard System Lockup / GPU Lost From Bus under Load - RTX 5070 (595.58.03) on Kernel 7.0.0 [Bug Report Attached]

  3. GitHub Issue #1080 - GSP heartbeat timeout
    RTX 5090 (GB202): GSP heartbeat timeout -> Xid 109/8 under Vulkan load via Proton (595.58.03, 590.48.01) · Issue #1080 · NVIDIA/open-gpu-kernel-modules · GitHub

Best regards,
Shakhizat

2

hello, @aplattner, Could you please help with this issue? I have posted in Discord channels and also sent an email to NVIDIA. It appears to be similar to: [580.105.08 Regression] Hibernate (S4) no longer powers off, fans stay on, resume still works

3

The GSP timeout is obviously caused by the GPU falling off the bus in the first place: how can it not time-out if the GPU is no longer there? ;-)

Now regarding falling off the bus, in the words of an NV eng:

So check these 3 things first and definitely the most recent driver (595.71 currently).
However if you browse this forum a bit, you will find that falling off the bus is something that Blackwell just tends to do for many ppl…

4

Hi, thanks for your reply, this might the same issue: RTX 5060 Ti (Blackwell) — GSP Firmware Crash causes GPU lockup (black screen, 100% fans, hard reset required) on nvidia-open 595.71.05 - Issues & Assistance - CachyOS Forum

5

I think also encountering this error or a related error with a 5070 Ti on an Asus TRX50 board. My system runs Windows, however symptoms are effectively identical:

  • the OS still responds to VNC connections (but show a black screen)
  • the GPU fully freezes showing a static image
  • the OS is unresponsive to any keyboard/mouse inputs
  • I’m unable to restore operation, requiring a power cycle to restore the system.

If I have telemetry/monitoring software displaying system metrics across the moment of failure, there are some patterns for things like PCIe bus interface speeds, bus utilisation, GPU +12V supply, power and fan speed. The crash/failure moment is otherwise ‘silent’, and happens under a variety of conditions where the GPU has a steady, medium workload (in my case NVENC/NVDEC). I’ve seen discussions where CUDA workloads have been involved.

My crashes have occurred between 4-6 days. The system previously benchmarked OK running the ‘usual’ industry benchmark tools for several days, so perhaps I didn’t leave the system benchmarking long enough to reveal the fault earlier.

A lack of relevant event logs or other telemetry suggests to me that the system is not kernel panicking, and the GPU drivers are not catching the fault - or perhaps they are causing the fault? Maybe it’s a GSP issue occurring so quickly that TDR isn’t able to handle it?

The fault has followed the 50 series GPU across two very different spec test systems using various releases of Studio drivers over the last few months. In the TRX50 system, after I installed an 30-series RTX GPU for testing, the fault has not reoccurred.

GPU and CPU thermals are fine, the PSU is intentionally massively overspecced for the system, all firmwares, drivers and BIOSes are latest stable versions.

6

if the fault follows the GPUs, they’re likely broken, at least one of them is

try running your workloads separately on each GPU for a couple days, maybe you’ll be able to isolate it