RTX PRO 6000 Blackwell (GB202) - Recurring full chip reset during sustained LLM inference, requires PSU power cycle to recover

Graphics / Linux Linux
1

My RTX PRO 6000 Blackwell Workstation Edition enters an unrecoverable fault state during sustained GPU compute (LLM inference via vLLM). The GPU operated 24/7 without issue for several months before this started occurring approximately one week ago. The issue has now occurred multiple times. Driver rollback does not resolve it.

This appears to match the known Blackwell GSP firmware crash pattern reported in these threads:

I am requesting assistance in confirming this is a hardware failure so I can proceed with RMA. NVIDIA customer support (chat) directed me to this forum as a prerequisite for the RMA process.

System Information

  • OS: CachyOS (Arch-based), kernel 6.19.9-1-cachyos, x86_64
  • CPU: AMD Ryzen 9 9950X (32 cores)
  • RAM: 125 GB
  • Motherboard: MSI MS-7E59 (2.0)
  • GPU 0: NVIDIA GeForce RTX 5090 (32GB) — Bus 01:00.0, display/desktop — no issues
  • GPU 1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (97GB) — Bus 03:00.0, compute only — faulting
  • Driver: 595.45.04 (NVIDIA UNIX Open Kernel Module)
  • CUDA: 13.2
  • GSP Firmware: 595.45.04
  • PCIe: Both slots Gen5 32GT/s x8 (board bifurcates x8/x8 with 2 GPUs — by design, confirmed in mobo manual)

Faulting GPU Identification

  • Product: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
  • Serial Number: 1792525051461
  • GPU UUID: GPU-37cb5368-58ab-0c04-75a8-d244d5811027
  • VBIOS: 98.02.52.00.02
  • Board Part Number: 900-5G144-2200-000
  • Device ID: 0x2BB110DE
  • Sub System ID: 0x204B10DE

Workload

vLLM inference server running a Mixture-of-Experts model (Qwen3.5-122B-A10B-NVFP4) with --gpu-memory-utilization 0.85. The server runs 24/7 serving API requests. This ran without issue for months before the failures began ~1 week ago.

Failure Behavior

  1. The GPU does not disappear from the system — it remains visible in nvidia-smi but reports ERR! for Fan, Power, GPU-Util, and MIG. GPU Recovery Action: Reset, Product Brand: GPU requires reset.
  2. The system does not freeze — the host OS, GPU 0 (RTX 5090), and all non-GPU 1 processes continue running normally.
  3. The application does not crash cleanly — the vLLM HTTP server stays alive (health endpoint responds), but the compute subprocess enters an infinite spin (process state R) because the GPU is unresponsive underneath it.
  4. No automatic recovery — the GPU remains faulted indefinitely. nvidia-smi -q -i 1 hangs. nvidia-smi -r -i 1 does not recover it. nvidia-bug-report.sh hangs because it queries the faulted GPU. A full PSU power cycle is required — soft reboot is not sufficient.

Kernel Log Evidence

At the moment of failure (2026-04-06 01:24:38), the kernel logged 74,341 register read errors in a 1-second burst, all on the same register address with the same poison value:

NVRM: GPU1 gpuHandleSanityCheckRegReadError_GH100: Possible bad register read:
  addr: 0xb830b0, regvalue: 0xbadf3200, error code: Unknown SYS_PRI_ERROR_CODE

Followed immediately by assertion cascades:

NVRM: GPU1 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:573
NVRM: GPU1 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
NVRM: GPU1 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375

Then repeating every ~1 second:

NVRM: GPU1 subdeviceCtrlCmdMcServiceInterrupts_IMPL: NVRM_RPC: NV2080_CTRL_CMD_MC_SERVICE_INTERRUPTS failed with error 0x62

Subsequent attempts to query the GPU produced:

NVRM: nvCheckOkFailedNoLog: Check failed: Reset required [NV_ERR_RESET_REQUIRED] (0x00000062)
  returned from _gpushareddataSendDataPollRpc @ gpu_user_shared_data.c:620

And nvidia-smi processes entered uninterruptible sleep (state D), blocked on uvm_va_space_destroy trying to release the dead GPU.

No GPU 1 errors appear in the kernel log prior to 01:24:38 this boot (booted 2026-04-04 13:02). The failure is sudden with no precursor warnings.

What Has Been Ruled Out

Possible cause Status Evidence
Thermal Ruled out GPU temp was 28°C at time of fault
VRAM OOM Ruled out 87,986 / 97,887 MiB used — within the 0.85 utilization target
Driver version Ruled out Issue occurs on both 580.x and 595.45.04
PCIe link degradation Ruled out x8 @ Gen5 32GT/s, correct for this motherboard’s bifurcation
Software/application bug Ruled out GPU registers returning poison value (0xbadf3200) — this is below the driver/application layer
GPU 0 interference Ruled out GPU 0 (RTX 5090) is healthy and unaffected during GPU 1 faults
PCIe slot Ruled out Swapped GPU to different PCIe slot — fault follows the card, not the slot

Similarity to Known Reports

My symptoms match the threads linked above precisely:

  • Same GPU: RTX PRO 6000 Blackwell (GB202), same VBIOS 98.02.52.00.02, same board part 900-5G144-2200-000
  • Same pattern: Stable for days/weeks under sustained compute, then sudden unrecoverable fault
  • Same error codes: Register poison reads, NV_ERR_GPU_IN_FULLCHIP_RESET, error 0x62 (timeout)
  • Same recovery requirement: Full PSU power cycle
  • Same CPU platform: Ryzen 9 9950X (the Xid 62/154 thread reports 9950X3D)
  • Driver rollback ineffective: Confirmed across 570, 575, 580, 595 by multiple reporters

In the Xid 119 thread, one user resolved this via RMA (request 904421), suggesting a defective unit.

Attachments

  • nvidia-bug-report.log.gz — collected in clean state after power cycle recovery (faulted-state collection not possible as the script hangs on the unresponsive GPU)
  • Full kernel logs and nvidia-smi captures from the faulted state available on request

Request

Please confirm whether this error pattern (register poison reads, fullchip reset, unrecoverable fault requiring PSU power cycle) is indicative of a hardware failure, so that I can proceed with the RMA process through NVIDIA customer support.

Thank you.
nvidia-bug-report.log.gz (657.6 KB)

2

I wrote one of those other threads. Run cudamemtest and gpuburn test. Cudamem passed but gpuburn failed each time. Its a firmware issue, I sent it back to PNY for RMA just recently. They will take some time to get back to me I guess. But yeah, I have two of these cards and one went faulty. Best of luck to you, I would also do the above tests and pack that up and open a case with PNY. Make sure you have your purchase receipt.

3

Thanks for the insight. I confirmed the predictable gpu_burn failure mode but it was consistent across both PCIE slots. However, after reducing the power limit from 600W to 450W I can run gpu_burn for 2 hours with no failures. I can’t see this being a power supply issue itself since I can run my other gpu at full wattage without issue. Will have to dig in a little further.

4

Oh yeah, thats different. I could only run gpuburn for a few minutes before it crashed. The wattage did not matter in this case (600W or 450W). Did you try and remove all other GPUs from the motherboard and check at 600W after that? Did you take the bug report after or before a restart post-crash?

5

Same issue here. Adding my report.
•GPU: RTX PRO 6000 Blackwell Workstation Edition
•Driver: 580.142 (open kernel modules)
•CUDA: 13.0
•Platform: Gigabyte TRX50 AI TOP / AMD Threadripper PRO 9965WX
•OS: Ubuntu 24.04.4 LTS, kernel 6.8.0-111-generic
•Workload: Ollama LLM inference, sustained 50+ minutes (qwen2.5-coder:32b-instruct-fp16, gpt-oss:120b)
•Power draw at crash: 460-480W sustained
•GPU temp at crash: 66°C (well within limits)
Crash behaviour: Hard system freeze, no Xid errors in kernel log, no dmesg output captured. Requires full PSU power cycle. Repeated crashes caused BIOS to stick at POST code A6, requiring CMOS clear.
Mitigations attempted:
•GSP firmware disable (NVreg_EnableGpuFirmware=0) — did NOT resolve
•IOMMU enabled, ACS enabled, AER enabled — did NOT resolve
•Display moved off PRO 6000 to separate GPU — did NOT resolve
•nvidia_drm modeset=1, nomodeset removed from GRUB — did NOT resolve
•Crash is reproducible at the same point in the same sustained workload
Power limiting to 400W is the only remaining workaround being tested.
This is affecting a production always-on AI inference server. Would appreciate any update on firmware fix timeline.

6

Are you perhaps using vllm or sglang with flashinfer cutlass attention backend to serve these models ?

7

No — I’m using Ollama (v0.20.3) for inference, not vLLM or SGLang. No FlashInfer or Cutlass attention backend. Standard Ollama serve with default settings.
The crash occurs during sustained inference generation (50+ minutes continuous) regardless of model — reproduced with both qwen2.5-coder:32b-instruct-fp16 (65GB, dense) and gpt-oss:120b (65GB, dense). Shorter inference runs (under 30 minutes) complete without issue.

8

Yea i think llamacpp are using the same kernels in the background (ollama uses llama.cpp) , because i keep having the same issue with flashinfer.

Switching to triton was the temporary solution till the drivers are fixed

9

Thanks think I’m going to try use SGLang with triton till NVIDIA can fix this, run some tests to see how I go.

10

Any updates from nvidia on this or reliable fixes for this ?

Getting annoying to deal with and its happening randomly now with anything basically doing heavy load work , not just llm inference.

11

I do have similar issues with llama.cpp or UNet model training…super annoying! I tried everything!