Xid 62 / Xid 154 GSP PMU halt crash on RTX PRO 6000 Blackwell during LLM inference

Graphics / Linux Linux
1

Continuing the discussion from Xid 62 / Xid 154 GSP PMU halt crash on RTX PRO 6000 Blackwell during LLM inference:

Xid 62 / Xid 154 GSP PMU halt crash on RTX PRO 6000 Blackwell during LLM inference

Summary

NVIDIA RTX PRO 6000 Blackwell Workstation Edition (GB202) crashes with Xid 62 followed by Xid 154 during LLM inference workloads. The GPU’s GSP firmware reports that the PMU has halted, requiring a full power cycle (soft reboot does not recover the GPU). This occurs on driver 595.58.03, which was supposed to address Xid 154 GSP crashes on Blackwell.

System Information

Component Details
GPU 2x NVIDIA RTX PRO 6000 Blackwell Workstation Edition (GB202)
VBIOS 98.02.52.00.02
Board Part Number 900-5G144-2200-000
Driver 595.58.03 (open kernel modules, .run installer)
CUDA 13.2
OS Ubuntu 24.04.4 LTS
Kernel 6.17.0-14-generic (x86_64)
CPU AMD Ryzen 9 9950X3D 16-Core
RAM 128 GB
PCIe GPU 0 at 01:00.0, GPU 1 at 03:00.0

Software Stack

  • vLLM 0.17.1
  • PyTorch 2.10.0+cu130
  • Model: Mixture-of-Experts model with FP8 (E=256, N=512, block_shape=[128,128])
  • Running with tensor parallelism across both GPUs (TP=2)
  • NCCL_P2P_DISABLE=1 (required for multi-GPU on this platform)

Crash Sequence (kernel log)

The crash occurs with no preceding warnings — GPU 0 goes from healthy to fully locked in under a second:

Mar 27 17:40:36 kernel: NVRM: GPU at PCI:0000:01:00: GPU-df022e84-9140-5762-5007-49bd93f72c3e
Mar 27 17:40:36 kernel: NVRM: GPU Board Serial Number: 1791625016090
Mar 27 17:40:36 kernel: NVRM: Xid (PCI:0000:01:00): 62, 223b6a66 00002100 00000000 20662342 20662ba6 20664272 2065f7ca 2063f48c
Mar 27 17:40:36 kernel: NVRM: GPU0 _kgspRpcGspEventPmuHalted: Received signal from GSP that PMU has halted.
Mar 27 17:40:36 kernel: NVRM: GPU0 nvCheckOkFailedNoLog: Check failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062) returned from pRmApi->Control(...) @ gpu.c:7442
Mar 27 17:40:36 kernel: NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)
Mar 27 17:40:36 kernel: NVRM: Xid (PCI:0000:01:00): 45, pid=12542, name=VLLM::Worker, channel 0x00000002

After this, the driver enters a loop of failed teardown attempts:

Mar 27 17:40:48 kernel: NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!  Notify Timeout Seconds: 7
Mar 27 17:40:51 kernel: NVRM: GPU0 kgmmuClientShadowFaultBufferUnregister_IMPL: Unregistering non-replayable fault buffer failed (status=0x00000062), proceeding...
Mar 27 17:40:51 kernel: NVRM: GPU0 kgmmuFaultBufferReplayableDestroy_IMPL: Unregistering Replayable Fault buffer failed (status=0x00000062), proceeding...

The RC watchdog continues firing every ~8 seconds indefinitely. nvidia-smi shows GPU 0 with ERR! status. The GPU does not recover without a full PSU power cycle.

History

  • Previously on driver 580.126.09, we experienced frequent Xid 154 GSP crashes (without the Xid 62 / PMU halt precursor).
  • Upgraded to 595.58.03 on 2026-03-27 specifically to address Xid 154 on Blackwell.
  • The crash still occurs, but now shows Xid 62 (PMU halt) as the root cause triggering Xid 154.
  • The same hardware runs stable with SGLang 0.5.9 using --attention-backend triton --fp8-gemm-backend cutlass flags, which avoid flashinfer and DeepGemm backends. This suggests certain GPU kernel dispatches may be triggering the firmware crash.

Steps to Reproduce

  1. Install driver 595.58.03 with open kernel modules on Blackwell RTX PRO 6000
  2. Run vLLM 0.17.1 with a large MoE model (FP8, TP=2)
  3. Serve inference requests — crash occurs during active inference (timing varies)

Expected Behavior

GPU should not experience firmware-level PMU halt during compute workloads.

Actual Behavior

GPU 0 GSP firmware reports PMU halt (Xid 62), triggering unrecoverable Xid 154. Full power cycle required to restore GPU.

nvidia-bug-report.log

[Attach nvidia-bug-report.log.gz — see below]