Deterministic Xid 120 GSP kernel panic on RTX PRO 6000 Blackwell under sustained cuBLAS workload (595.84)

Graphics / Linux Linux
1

## System

- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB)

- Driver: 595.84 (also reproduced on 580.126.09 with different signature, see below)

- OS: Ubuntu, kernel 6.17.0-35-generic (also reproduced on 6.8.1-1042-realtime PREEMPT_RT)

- CUDA 13.x, PyTorch (cu130 wheels), persistence mode ON

## Symptom

Under sustained fp32 batched-GEMM-heavy compute (PyTorch, `cublasSgemmStridedBatched`-dominated,

batch ~4096 candidates x ~20k grid points, einsums of 3x3 matrices, continuous ~100% utilization),

the GSP crashes after minutes-to-hours. The application receives

`CUDA error: CUBLAS_STATUS_EXECUTION_FAILED`; afterwards `nvidia-smi` shows `ERR!`/`[N/A]`

fields and `torch.cuda.is_available()` returns False until reboot.

## Key evidence: deterministic GSP kernel panic (Xid 120)

Every crash on 595.84 produces an identical GSP-CrashCat report — same pc, same condition,

same aux, same scause/stval, across independent runs and reboots:

NVRM: Xid (PCI:0000:01:00): 120, GSP kernel panic: condition failed (11)

      @ pc:0xffffffff9300dae6, aux:0xe2, partition:4#0

Reported by libos partition:4#5 kernel v3.1

sepc:0xffffffff9300299a  stval:0xffffffff70000d08  scause:0xf (store/AMO page fault)

GSP bin buildId: 6332050fe14352839619385982f83700835ef570

A physically degrading part would fail at varying locations; a fixed panic PC across

many independent crashes points to a firmware code-path defect.

## What we ruled out experimentally

- Kernel: crashes on both PREEMPT_RT 6.8 and generic 6.17.

- Driver: 580.126.09 fails differently (Xid 62 → Xid 45 → repeated Xid 119 GSP RPC

timeout → Xid 154 reset required); 595.84 fails with the Xid 120 panic above.

- Power limit: capping 600W → 400W (`nvidia-smi -pl 400`) extends time-to-crash

(~7 min → ~35 min) but does not eliminate it — consistent with a clock/power state

machine trigger, not with insufficient PSU capacity.

- No NVML queries during load; no process kills; persistence ON; reduced batch sizes;

ECC log clean; ambient temperature normal (GPU ~75 C at crash, within spec).

- The same workload ran 12.5 h continuously overnight without incident (same code/driver),

so trigger frequency is workload/state dependent, not time-deterministic.

## Repro

PyTorch loop evaluating batched 3x3 einsums + elementwise kinematics over

(4096 x ~20000) fp32 tensors, ~100% SM utilization, single process.

Open-source script and exact seeds available on request; time-to-crash varies from

minutes to hours depending on power limit / workload shape.

nvidia-bug-report.log.gz attached (captured immediately after a crash, before reboot).

2

Hi @ttopeor

We have seen similar issue internally and is currently being investigated via internal bug 6426268.

3

I am seeing a reproducible Xid 120 on Debian 13.

The task is running LM-Studio with Qwen3.8-27B and using a specific prompt.

## System

- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB)

- Driver: 610.57.04 (also reproduces on 595.91.07)

- OS: Debian 13, Ubuntu, kernel 6.12.101+deb13-amd64

- CUDA 13.x

4

Another datapoint for internal bug 6426268, with what may be a useful twist: on our RTX PRO 6000 Blackwell Workstation Edition the GSP hangs while the card is idle, not under load, and we have an exact, repeating RPC-level fingerprint from nvidia-bug-report.sh captures taken while the GPU was still wedged.

System

  • RTX PRO 6000 Blackwell Workstation Edition — Dell-channel OEM board, part 900-5G144-0100-000, serial 1793725035411, VBIOS 98.02.52.00.03, PCI 0000:41:00.0
  • Ubuntu 24.04, kernel 6.8.0-138-generic
  • Drivers: 595.84 (first hang), then 610.57.04 open kernel modules (hangs 2–4)
  • Workload: vLLM LLM inference. A second RTX PRO 6000 (different board SKU, 900-5G144-2200-000) in the same host has never failed.

Four GSP hangs of the same card in 9 days

  1. 2026-08-17 — driver 595.84, ~33 min after a sustained-load run ended, card idle at 37 °C
  2. 2026-08-20 — 610.57.04, idle
  3. 2026-08-24 — 610.57.04, idle (only healthcheck traffic), 37 min into a fresh boot
  4. 2026-08-25 — 610.57.04, idle overnight, ~10 h after the previous recovery

Two differences from this thread’s headline report: all four struck with the GPU idle or near-idle — it survives sustained LLM inference fine — and we get no Xid 120 / GSP-CrashCat panic; the GSP simply goes silent, followed ~75 s later by Xid 175 (RPC timeout) then Xid 154 (recovery action → PF FLR).

Identical death fingerprint in hangs #3 and #4

In both captures the GSP services a burst of client-teardown FREEs and internal GSP_RM_CONTROLs normally (237–1036 µs each), then never answers the immediately following GSP_RM_ALLOC (function 103, paramsSize 0x78 — an RM client allocation from a routine nvidia-smi invocation) issued ~137 µs later. GSP-RM and LibOS heartbeats stop at the same instant. The RPC event history shows nothing from the GSP since boot-time GSP_LOCKDOWN_NOTICEs — no warning or error event precedes the hang.

Hang #4 (2026-08-25):

NVRM: Xid (PCI:0000:41:00): 175, pid=664048, name=nvidia-smi, Timeout after 75s of waiting for RPC response from GPU0 GSP! Expected function 103 (GSP_RM_ALLOC) sequence 1232201 (0x0 0x78).
NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) sequence 1232202 and data 0x0000000020800a56 0x000000000000005c.
NVRM: GPU0 RPC history (CPU -> GSP):
NVRM:     entry function                     sequence data0              data1              ts_start           ts_end             duration actively_polling
NVRM:      0    76   GSP_RM_CONTROL           1232202 0x0000000020800a56 0x000000000000005c 0x000659d475f8c9cf 0x0000000000000000          y
NVRM:     -1    103  GSP_RM_ALLOC             1232201 0x0000000000000000 0x0000000000000078 0x000659d471805a2f 0x0000000000000000
NVRM:     -2    10   FREE                     1232200 0x00000000c1d15477 0x0000000000000000 0x000659d4717e40be 0x000659d4717e425b    413us
NVRM:     -3    10   FREE                     1232199 0x00000000a55a0040 0x0000000000000000 0x000659d4717e3f1c 0x000659d4717e40b2    406us
NVRM:     -4    10   FREE                     1232198 0x00000000a55a0050 0x0000000000000000 0x000659d4717e3d9e 0x000659d4717e3ef9    347us
NVRM:     -5    76   GSP_RM_CONTROL           1232197 0x0000000020800aff 0x0000000000000010 0x000659d4717e3c09 0x000659d4717e3d8a    385us
NVRM:     -6    10   FREE                     1232196 0x00000000a55a0090 0x0000000000000000 0x000659d4717e38cd 0x000659d4717e3ad5    520us
NVRM:     -7    76   GSP_RM_CONTROL           1232195 0x0000000020800aff 0x0000000000000010 0x000659d4717e2ac2 0x000659d4717e2c5a    408us
NVRM: GPU0 _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 912516671 heartbeat 38563595 heartbeatWithOffsetMs 912437285 diff 79386 timeout 5200
NVRM: GPU0 _kgspRpcRecvPoll: GSP RM heartbeat timed out
NVRM: GPU0 _kgspRpcRecvPoll: LibOS heartbeat timed out
NVRM: GPU0 _issueRpcAndWait: rpcRecvPoll timedout for fn 103 sequence 1232201!
NVRM: GPU0 rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1d15478; hParent=0x00000000; hObject=0x00000000; hClass=0x00000000; paramsSize=0x00000078; paramsStatus=0x0000005c; status=0x00000065
NVRM: Xid (PCI:0000:41:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)

Hang #3 (2026-08-24) — same shape, down to the pending follow-up control (0x20800a56, 0x5c bytes) in the buffer:

NVRM: Xid (PCI:0000:41:00): 175, pid=1016305, name=nvidia-smi, Timeout after 75s of waiting for RPC response from GPU0 GSP! Expected function 103 (GSP_RM_ALLOC) sequence 9980409 (0x0 0x78).
NVRM: GPU0 GSP RPC buffer contains function 76 (GSP_RM_CONTROL) sequence 9980410 and data 0x0000000020800a56 0x000000000000005c.
NVRM: GPU0 RPC history (CPU -> GSP):
NVRM:     entry function                     sequence data0              data1              ts_start           ts_end             duration actively_polling
NVRM:      0    76   GSP_RM_CONTROL           9980410 0x0000000020800a56 0x000000000000005c 0x000659c5e67e3502 0x0000000000000000          y
NVRM:     -1    103  GSP_RM_ALLOC             9980409 0x0000000000000000 0x0000000000000078 0x000659c5e205c5f2 0x0000000000000000
NVRM:     -2    10   FREE                     9980408 0x00000000c1dab4a9 0x0000000000000000 0x000659c5e1f5ca43 0x000659c5e1f5cb30    237us
NVRM:     -3    10   FREE                     9980407 0x00000000a55a0040 0x0000000000000000 0x000659c5e1f5c8fa 0x000659c5e1f5ca37    317us
NVRM:     -4    10   FREE                     9980406 0x00000000a55a0050 0x0000000000000000 0x000659c5e1f5c69a 0x000659c5e1f5c8df    581us
NVRM:     -5    76   GSP_RM_CONTROL           9980405 0x000000002080852f 0x000000000000034c 0x000659c5e1f5bfbe 0x000659c5e1f5c3ca   1036us
NVRM:     -6    103  GSP_RM_ALLOC             9980404 0x0000000000002080 0x0000000000000004 0x000659c5e1f5a78e 0x000659c5e1f5a922    404us
NVRM:     -7    103  GSP_RM_ALLOC             9980403 0x0000000000000080 0x0000000000000038 0x000659c5e1f5a59b 0x000659c5e1f5a72d    402us
NVRM: Xid (PCI:0000:41:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)

Possibly useful for your engineers: the driver’s GSP core-dump path fails on this card. In both captures kflcnCoreDumpDestructive_IMPL reports the core booted but then:

NVRM: GPU0 kflcnCoreDumpDestructive_IMPL: ICD: Core is booted.
NVRM: GPU0 kflcnCoreDumpDestructive_IMPL: ICD: RSTAT0 0x0000000000000004
NVRM: GPU0 kflcnCoreDumpDestructive_IMPL: ICD: [ERROR] ICD Halt command failed.

— the RISC-V core will not halt, so no capture from this card can ever include a GSP core dump.

Suspected trigger: sustained RM-client churn. The host’s monitoring tooling was execing nvidia-smi ~2.2×/s, sustaining ~30 RPC/s of RM client alloc/free against this GPU (RPC sequence numbers: 9.98 M over ~3.8 days on the hang-#3 boot; 1.23 M over ~10.7 h on the hang-#4 boot). Working hypothesis: a GSP-RM race in client free→alloc handling, hit probabilistically under constant churn — which would explain why it dies idle (churn dominates when nothing else is running) and survives heavy inference. We eliminated the churn on 2026-08-25 (persistent NVML session instead of subprocess polling); the recurrence interval from here will be informative either way.

Ruled out: thermal (hang #1 struck at 37 °C, 33 min after load ended; BMC SEL empty), PCIe (AER clean, full x16), memory (zero remapped rows), power/VRAM pressure (hang #4 occurred with a 300 W cap and 0.65 memory utilisation already applied).

Recovery: nvidia-smi -r → Not Supported on this SKU; a warm reboot makes it worse — the latched state survives and the kernel hangs loading the nvidia module from the initramfs. Only a full power cycle recovers it.

Related public reports that look like the same family: open-gpu-kernel-modules issues #1111 and #1259.

I have two nvidia-bug-report.log.gz files captured while wedged, pre-power-cycle: 2026-08-24 (1.9 MB, partial — the script itself hangs querying the dead GPU) and 2026-08-25 (2.98 MB, truncated by our capture timeout, but both RPC post-mortems above are in the recovered portion), plus nvidia-smi -q output. Attaching them here; happy to provide them through another channel if that’s easier for attaching to 6426268.
nvidia-bug-report-20260824-final.log.gz (1.8 MB)
nvidia-bug-report-20260825.log.gz (2.8 MB)
nvidia-smi-q-20260824.txt (28.4 KB)

5

I reproduced the GSP crash again on a new software stack.

System:

  • GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
  • Driver: 610.57.04, open kernel modules
  • Kernel: 7.0.0-30-generic
  • CUDA Toolkit: 13.3
  • GPUStack: 2.2.3

Workload:

  • qwen3.5-27B-audiofinal, GGUF BF16
  • llama.cpp build 8390 (b6c83aad5)
  • Single GPU, 131072 context, Flash Attention enabled
  • CUDA Graph and fused Gated Delta Net enabled

Timeline (Asia/Tokyo):

  • 2026-08-26 22:50:59: second inference started
  • 22:51:04: Xid 62, GSP PMU halted
  • 22:51:04: Xid 154, recovery action changed to PF FLR
  • 22:53:04: Xid 120, GSP kernel panic timeout

After the failure, Docker still reported the runner as running, but nvidia-smi returned N/A for GPU utilization and process information.

This shows the failure is not limited to Qwen3.8, SGLang, or FP8. Please link this reproduction to internal bug 6426268 and advise whether a driver/firmware fix or hardware RMA is appropriate.

The NVIDIA bug report and complete incident bundle are attached.

Incident bundle SHA256:
c11f7d2fa4a865c766c32eabe67513b8ef6c832a7f1e18c504da109b43f1cc66

nvidia-crash-20260826-225606.ONkcFh.tar.gz (1.4 MB)

nvidia-bug-report-20260826-225606.log.gz (371.4 KB)

6

5th occurrence — and it eliminates our own leading hypothesis: the GSP hang recurs with the RM-client churn removed, on an idle card.

Following up on my post above (occurrences 1–4). We have now had a fifth GSP hang on the same card (RTX PRO 6000 Blackwell Workstation Edition, Dell-channel board 900-5G144-0100-000, 41:00.0, driver 610.57.04 open kernel modules, GSP firmware). New bug report attached — 3.5 MB, our largest capture yet (the script still never completes against the dead GPU; killed at a 900 s timeout).

The key new data point: after occurrence #4 we identified and eliminated the ~30 RPC/s RM client alloc/free churn described in my previous post (a telemetry service exec’ing nvidia-smi ~2.2×/s was rewritten onto a single persistent NVML session; verified — that host’s nvidia-smi exec rate went to ~0, and this boot’s Xid sequence number implies roughly an order of magnitude lower RPC rate). The hang recurred anyway, ~2.7 days into the clean boot, on an idle card (the only GPU activity was a 30-second-interval single-token healthcheck completion from vLLM). So sustained client free→alloc churn is at most an accelerant — the interval between failures stretched from hours back to days when we removed it — but it is not required. That is 5 of 5 occurrences now with the card idle at onset.

Fingerprint details for this one (2026-08-28, times UTC):

  • 00:35:13 — last successful inference (healthcheck completion)
  • 00:36:46 — Xid (PCI:0000:41:00): 175, Timeout after 75s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 906774 (0x2080a61d 0xc)
  • 00:36:50 — _kgspIsHeartbeatTimedOut (diff ~81 s vs 5.2 s timeout), then Xid (PCI:0000:41:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)
  • aftermath — with the telemetry churn gone, the log is no longer a 1–2 M line/min NV_ERR_RESET_REQUIRED flood: just an assert pair (kbusSendBusInfo → NV_ERR_RESET_REQUIRED at kern_bus_ctrl.c:257, plus rs_client.c/rs_server.c asserts) every ~900 s from residual monitoring.

Two differences from occurrences #3/#4 worth flagging:

  1. The RPC the GSP died on this time was a GSP_RM_CONTROL (function 76, cmd 0x2080a61d, paramsSize 0xc), not the GSP_RM_ALLOC (function 103, 0x78) client allocation that both #3 and #4 hung on. Same silent-death behaviour (no error event from the GSP, heartbeats stop, nothing in the RPC event history), different victim RPC — consistent with the GSP itself dying and the logged RPC merely being whoever asked next.
  2. Occurrence #4 onset was 00:47:12 UTC and #5 was 00:36:46 UTC — two of five in the same ~00:35–00:47 UTC window, both idle. We have no scheduled GPU work at that time and no explanation; noting it in case it means something firmware-side (e.g. a periodic internal task).

On the Xid 154 → PF FLR recovery hint: we intended to test a driver-mediated FLR before power-cycling this time, but couldn’t — the vLLM process was stuck unkillable in D-state against the dead GPU, so the device could never be quiesced for a reset. Recovery again required a full chassis power cycle (a warm reboot reliably hangs at kernel handoff with the latched GPU, as reported previously). If there is a supported way to attempt the PF FLR recovery with an uninterruptible process still holding the device — or any GSP-side debug/logging you would like enabled before the next occurrence — we will gladly run it; this card reproduces every few days.

Also repeating from the #4 analysis in case it was missed: the driver’s own GSP core-dump path fails on this card (kflcnCoreDumpDestructive: ICD Halt command failed — the RISC-V core won’t halt), so no GSP core dump exists in any of our captures.

Environment recap: Ubuntu, driver 610.57.04 (open, GSP), vLLM under Docker; failures have now occurred across three engine image generations (v0.22.1, v0.27.1, v0.28.0 CUDA 12.9), two driver branches (595.84, 610.57.04), under load and — mostly — idle. Same physical card every time (slot placement verified by UUID against pre-incident records); we are planning a slot swap to finally discriminate card vs slot, and expect to replace the card. Any movement on internal bug #6426268 would be very welcome.

7

Any update on this? I just had one of my 6000’s start doing this every 20 hours or so.

8

I would like to add another RTX PRO 6000 Blackwell Workstation Edition failure that may be related to internal bug 6426268, although my
board has now progressed from a runtime GSP failure to a persistent GSP-FMC initialization failure.

Hardware and software

  • GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB
  • Board SKU: 900-5G144-2200-000
  • Model: PG144C
  • Assembly: 699-5G144-0520-B00
  • Serial: ending 6519
  • PCI device: 10de:2bb1
  • VBIOS printed on board: 98.02.81.00.07
  • OS: Ubuntu 24.04
  • Kernel: 6.8.0-138-generic
  • Drivers tested: 595 branch and 610.43.02, open kernel modules
  • Workload: vLLM inference using tensor parallelism across four RTX PRO 6000 Blackwell GPUs

Original runtime failure

The same physical GPU repeatedly became the failed/ERR! GPU during the multi-GPU workload. The original incident included:

Xid 62 — GSP/PMU halted
Xid 154 — GPU recovery/reset required
Xid 45 — associated application termination

After the failure, the affected GPU was no longer usable through NVML/CUDA, displayed ERR in nvidia-smi and required a system restart or complete power cycle.

Isolation testing

The failing GPU was initially at PCI address 50:00.0. I swapped it with another identical, known-working RTX PRO 6000.

After the swap:

  • The known-working card operated correctly in the former failing slot.
  • The failure followed the physical board ending in serial 6519.
  • The affected card enumerated normally through PCIe but no longer appeared in nvidia-smi.

I subsequently tested the affected board:

  • In multiple PCIe slots
  • On the 595 and 610.43.02 driver branches
  • After a complete shutdown and removal of AC power
  • Back in its original slot
  • Directly in the motherboard as the only installed GPU
  • With the external Switchtec/C-Payne PCIe switch completely removed from the path

The result remained identical in every configuration.

Current persistent failure

The motherboard still enumerates the card:

0000:4c:00.0 VGA compatible controller [0300]:
NVIDIA Corporation Device [10de:2bb1] (rev a1)

The PCIe link trains successfully at Gen5 x16. PCIe configuration space remains readable, and I found no AER, DPC, lane-error, or
Switchtec error counters associated with the failure.

However, NVIDIA driver initialization now fails repeatedly:

NVRM: GPU0 gpuReportGspFmcErrorCode_GH100:
Fatal GSP-FMC Error: version=0x1, partition=0x1,
error code=0xaff, additional info=0xb

NVRM: GPU0 kgspInitRm_IMPL:
Max GSP-RM boot attempts exceeded: 4/4

NVRM: GPU0 RmInitAdapter:
Cannot initialize GSP firmware RM

NVRM: GPU 0000:4c:00.0:
RmInitAdapter failed! (0x62:0x55:2119)

NVRM: GPU 0000:4c:00.0:
rm_init_adapter failed, device minor number 0

Consequently:

$ nvidia-smi
No devices were found

The NVIDIA modules load successfully and report version 610.43.02; only GPU initialization fails.

Comparison with working boards

The other identical RTX PRO 6000 boards continue to operate correctly in the same host, slots, power configuration, and software
environment.

A full lspci -vvv comparison found:

  • Clean Gen5 x16 link on the affected card
  • Zero correctable and uncorrectable AER status
  • Zero lane-error status
  • No PCI status faults
  • MSI remains disabled/unconfigured on the affected card because the NVIDIA driver never completes initialization

This behavior follows the physical board and not its slot, PCI address, power cable, switch port, or driver installation.

9

Hi @sentdex_nv, your case may be distinct from the Xid 120 runtime GSP crash because you now have a persistent initialization failure, which could point to a hardware issue.

Do you have a known-good host or system where you can test this board? Please try the failing board in that known-good system. If it fails there as well, please capture an NVIDIA bug report and attach it here. This card may need to be replaced.

Hi @simon328, thank you for the log files and detailed information. Yes, please share your results after the board/slot swap, including whether the failure follows the GPU.

Hi @park_li, @abanu_capsunel, and @officiallymarky, could you please share the steps you use to run your workload? Please include the exact commands so we can match the same commands and load in our internal testing. Also, please attach an NVIDIA bug report if you have not already and let us know whether the failure happens on the same RTX PRO 6000 each time.

Is there a last known-good driver that did not hit this failure? Thank you.

10

I am in the same boat as Sentdex above, I just use Arch with latest nvidia drivers and vllm to run deepseek v4 flash. I am too scared to swap slots with the gpu and have the gpu completely unusable. Right now it gets about 10-20 hours before it goes into wedge state and I got to reboot. It’s always the same gpu and started 4-5 days ago. Exact same symptoms Sentdex had before the card completely locked.

11

Slot-swap result, as requested in post #9: the failure FOLLOWED THE GPU. Sixth occurrence, same board, now in the other slot.

@abchauhan — following up on your question about the board/slot swap.

On 29 Aug we physically exchanged the two RTX PRO 6000 Blackwell Workstation Edition boards between their slots (full AC drain, both boards and PCIe power cables reseated), so the board that had failed five times (Dell-channel 900-5G144-0100-000, serial 1793725035411, VBIOS 98.02.52.00.03, GPU-e39f357f-9891-6768-a2b8-77289ca9ac28) moved from 41:00.0 to 01:00.0, and the never-failed board (900-5G144-2200-000, VBIOS 98.02.81.00.07) moved from 01:00.0 to 41:00.0. Workloads were pinned so each board kept serving the same model.

On 4 Sep at 03:17 UTC the GSP hang recurred — at 01:00.0, i.e. on the same physical board in its new slot:

NVRM: GPU at PCI:0000:01:00: GPU-e39f357f-9891-6768-a2b8-77289ca9ac28
NVRM: GPU Board Serial Number: 1793725035411
NVRM: Xid (PCI:0000:01:00): 175, Timeout after 75s of waiting for RPC response from GPU1 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 837109 (0x2080a61d 0xc).
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (PF FLR)

The other board has now run 5.5 days in the slot where the failing board had died five times, with zero NVRM errors. So across six occurrences the fault tracks the board, not the slot, the PCIe root port, the power cabling or the host — consistent with @sentdex_nv’s finding in post #8 that his failure also followed the physical board. We are treating this board as faulty and I fear we will have to scrap it if we cannot get a VBIOS or firmware update that addresses this issue (it was bought second-hand, so there is no warranty route); the new bug report is attached in case the pattern is still useful to the #6426268 investigation.

Fingerprint (2026-09-04, UTC):

  • ~03:16:42 — GSP stops responding (reconstructed from _kgspIsHeartbeatTimedOut ... diff 45568 timeout 5200 logged at 03:17:28), followed by the usual gpuHandleSanityCheckRegReadError_GH100 ... 0xbadf3200 reads.
  • 03:17:59 — immediately before the GSP Timeout banner the driver also logged kgmmuInvalidateTlb_GM107: TLB invalidation failed waiting for prior invalidate (status=0x00000065) → kgmmuCheckMemSubsysError_IMPL: Memory Subsystem Error detected → _kgspCheckIfMemsysError: Unexpected error ... 0x72. We can’t tell whether that is cause or a consequence of the dead GSP (register reads were already returning the bad-read sentinel), but it is new relative to our earlier posts.
  • 03:17:59 — Xid 175 on GSP_RM_CONTROL seq 837109, cmd 0x2080a61d, 0xc bytes — the exact same control command that occurrence #5 died on (seq 906774 on 28 Aug). Follow-up RPC in the buffer at dump time is again 0x20800a56 (0x5c), as in #3/#4/#5.
  • RPC history: the preceding entries are a steady alternation of 0x2080a61d (0xc) and 0x2080852f (0x34c) controls completing normally in 220–604 µs, spaced ~2 s apart — that is a single persistent NVML monitoring client polling at 2 s (Comm: python3). Nothing from the GSP in the event history since the boot-time GSP_LOCKDOWN_NOTICEs. No warning, no error event — the GSP just stops answering.
  • kflcnCoreDumpDestructive_IMPL: ICD: Core is booted ... [ERROR] ICD Halt command failed — the RISC-V core still won’t halt, so again no GSP core dump. riscvPrivErrInfo: badf1002 in the non-destructive dump.
  • 03:18:03 — Xid 154 → PF FLR. Aftermath: quiet kbusSendBusInfo → NV_ERR_RESET_REQUIRED assert every ~900 s. Host otherwise idle (only a 30 s single-token healthcheck completion was running).

Interval since the post-swap boot: 5 d 12 h, idle at onset (6 of 6 now). Both boards were at a 400 W power cap; host-side RM client churn was ~0 (all monitoring on persistent NVML sessions) — neither changed anything.

Exact workload command, since you asked the other posters for theirs. This is the container on the failing board (Docker, vllm/vllm-openai:v0.28.0, CUDA_VISIBLE_DEVICES=0):

vllm serve --model /models/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b-fp8 \
  --host 0.0.0.0 --port 8013 --max-model-len 262144 --max-num-seqs 128 \
  --gpu-memory-utilization 0.65 --no-enable-prefix-caching --enable-chunked-prefill \
  --max-num-batched-tokens 8192 --mamba-ssm-cache-dtype float16 --mamba-cache-dtype float16 \
  --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num_speculative_tokens":4}'

Host: Ubuntu, kernel 6.8.0-138-generic, driver 610.57.04 open kernel modules (GSP), ASUS Pro WS WRX80E-SAGE SE WIFI, BIOS 1801. The same board also failed on 595.84 and across vLLM v0.22.1 / v0.27.1 / v0.28.0 images. To your last question — no, we have no last known-good driver: this board has failed on every branch we have run it on (595.84 ×2, 610.57.04 ×4), while the sibling board has never failed on either.

One practical note for others hitting this: on this host nvidia-bug-report.sh has always hung against the dead GPU and had to be killed on a timeout; today’s run completed for the first time (3.4 MB, 15 min), so it is worth letting it run rather than assuming it is stuck.

Recovery still needs a full chassis power cycle (a warm reboot hangs at kernel handoff with the latched GPU; a driver-mediated PF FLR isn’t reachable because the inference process is left uninterruptible in D-state against the dead device).
nvidia-bug-report-20260904.log.gz (3.3 MB)
nvidia-smi-q-20260904.txt (28.3 KB)

12

Seventh occurrence, same board, 32 hours after the last power cycle. Fingerprint identical to #5 and #6 — same GSP_RM_CONTROL command, same GSP program counter.

Short update for the record, since this one adds repetition rather than anything new.

The board that failed six times before (Dell-channel 900-5G144-0100-000, serial 1793725035411, GPU-e39f357f-9891-6768-a2b8-77289ca9ac28, still at 01:00.0 after the 29 Aug slot swap) hung again on 5 Sep at 17:01 UTC, 1 d 7 h 49 m after the boot that followed occurrence 6. The sibling board at 41:00.0 is now at 8 days in the slot where this board died five times, still with zero NVRM errors.

Fingerprint (2026-09-05, UTC):

  • ~17:01:31 — GSP stops responding (reconstructed from _kgspIsHeartbeatTimedOut ... diff 45149 timeout 5200 at 17:02:16), then the gpuHandleSanityCheckRegReadError_GH100 ... 0xb830b0 / 0xbadf3200 flood.
  • 17:02:47 — kgmmuInvalidateTlb_GM107: TLB invalidation failed waiting for prior invalidate (status=0x00000065) → Memory Subsystem Error detected → _kgspCheckIfMemsysError: ... 0x72 — the same preamble as occurrence 6.
  • 17:02:47 — Xid 175, Timeout after 75s of waiting for RPC response from GPU1 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 202541 (0x2080a61d 0xc). That is the third consecutive hang that died on control 0x2080a61d (0xc bytes) — occurrence 5 seq 906774, occurrence 6 seq 837109, now seq 202541. Follow-up RPC in the buffer at dump time is 0x20800a56 (0x5c) again, as in 3/4/5/6.
  • RPC history immediately before: the usual 2 s alternation of 0x2080a61d (0xc) and 0x2080852f (0x34c) from the single persistent NVML monitoring client, every one completing normally in 191–614 µs, then the dead one. Event history (GSP → CPU) still shows nothing since the boot-time GSP_LOCKDOWN_NOTICEs — no warning event precedes the silence.
  • kflcnCoreDumpNondestructive_IMPL: PRI: riscvPc : 01aa13aa, riscvIrqmask : 810400d2, riscvPrivErrInfo : badf1002, fbifCtl / fbifThrottle : badf5720. riscvPc 0x01aa13aa is now the parked PC in four of our five captured hangs (24 Aug, 28 Aug, 4 Sep, 5 Sep; only 25 Aug differed at 0x9300650e). ICD Halt command failed again, so still no destructive core dump.
  • 17:02:51 — Xid 154 ... recovery action ... PF FLR. Aftermath as before: kbusSendBusInfo → NV_ERR_RESET_REQUIRED assert whenever anything touches the device.
  • PCIe AER on 01:00.0 and root port 00:01.1: UESta/CESta all clear, DevSta no errors — clean, as in every capture.

Conditions at onset: host idle apart from a 30 s single-token healthcheck completion and the 2 s NVML poll; both boards at a 400 W cap; ~45 °C. Same host, driver 610.57.04 open kernel modules, kernel 6.8.0-138-generic, vllm/vllm-openai:v0.28.0 with the vllm serve command in post #10 (unchanged).

Intervals between hangs on this board so far: ~2 d, ~4 d, ~10 h, ~3 d, 5.5 d, 1.3 d (measured from each preceding recovery boot) — no trend we can see; workload, power cap, slot and host-side RM churn have all been varied without effect.

We have decided not to replace the board for now (its only workload is behind an automatic fallback, so these hangs cost us nothing but a power cycle), which means we can keep supplying captures if they are useful. If there is anything specific you would like enabled or collected on the next occurrence — an RM debug level, a particular nvidia-debugdump invocation, a firmware trace setting — tell us and we will have it in place; at the current rate the next one is likely within the week.

Recovery unchanged: full chassis power cycle (warm reboot hangs at kernel handoff with the latched device; the inference process is left in D-state so a driver-mediated FLR is not reachable). 46 s to POST, both boards clean on return.
nvidia-bug-report-20260906.log.gz (2.1 MB)

13

Hi @abchauhan

I ran a few tests on my system and attach the bug reports and instructions that reproduce the issue below.

Here are the steps:

  • have Debian 13 system with RTX 6000 PRO Blackwell Workstation card
  • install 610.57.04 NVIDIA driver
  • install LM-Studio from lmstudio_dot_ai/download
  • open LM-Studio
    • download the Qwen3.8 27B model from here lmstudio_dot_ai/models/qwen/qwen3.8-27b
    • load the model with default settings
  • go to chat tab, start new chat (see attached screenshot)
  • enter the following prompt: “create an objective-c application to test text to speech by providing an entry area for text and a way to select available voices and a button to start the voice.”
  • submit the prompt
  • observe how the NVIDIA driver crashes within 1-2 minutes

The same crash occurs when I limit the power with nvidia-smi:

  • 400W
  • 200W

The same crash occurs with both:

  • 610.57.04 driver
  • 580.76.05 driver
  • not sure of a driver where this does not occur

The same crash occurs with different versions of the Qwen3.8 model:

  • lmstudio_dot_ai/models/qwen/qwen3.8-27b
  • huggingface_dot_co/unsloth/Qwen3.8-27B-GGUF

No crash occurs with the Qwen3.6 model (previous version):

  • lmstudio_dot_ai/models/qwen/qwen3.6-27b

I hope this helps.

nvidia-bug-report_driver580.log.gz (739.0 KB)

nvidia-bug-report_driver610_unsloth.log.gz (913.5 KB)

Screenshot From 2026-09-15 16-57-39
Screenshot From 2026-09-15 16-57-391582×1035 59.1 KB

nvidia-bug-report_driver610.log.gz (770.6 KB)

14

Hi @capsunel , Bug with ID 6818900 raised for tracking this issue.

15

Thank you @abanu_capsunel and @simon328 for the additional information.
We are filing multiple bugs to track the issues reported here.

@simon328 - Could you please share which vendor/OEM this particular card was purchased from?

@ttopeor - Your Xid 120 GSP kernel panic on RTX PRO 6000 BW under sustained cuBLAS workload is being tracked on internal bug #6699452.

16

This is a Dell card, part 900-5G144-0100-000, serial 1793725035411, VBIOS 98.02.52.00.03

17

Hi @officiallymarky

Bug 6426268 turned out to be different issue and have different root cause.

We are filing multiple bugs to track the issues reported here.

19

Hi @simon328

I have filed a bug 6819883 for tracking purpose. I will try to duplicate issue locally and will get back if required any additional information.

20

Hi @sentdex_nv

I have filed a bug 6821987 for tracking purpose. I will try to duplicate issue locally and will get back if required any additional information.

21

Do we know what year these bugs will be fixed?