# RTX PRO 6000 Blackwell WS: whole-host hard lock during vLLM CUDA-graph

**URL:** <https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239>\
**Category:** CUDA Programming and Performance\
**Tags:** rtx\
**Created:** [August 5, 2026, 11:38am UTC](https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239 "2026-08-05T11:38:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gpersoon](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/gpersoon/32/497240_2.png) [@gpersoon](https://forums.developer.nvidia.com/u/gpersoon)\
**Post date:** [August 5, 2026, 11:38am UTC](https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239/1 "2026-08-05T11:38:49Z")

</div>

My entire PC was crashing on a specific vLLM load - i was able to reproduce it, see below.  
Can anyone else reproduce it ? (if not it might be related to my setup)  
Also reported here: [Log in | NVIDIA Developer](https://developer.nvidia.com/bugs/6539136)  
Details:  
GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition  
VBIOS: 98.02.81.00.07  
Driver: 610.43.03 open kernel module, CUDA 13.3  
Host: ASUS ROG STRIX X870E-E, Ryzen 9 9950X3D, 64 GiB DDR5-4800  
OS/kernel: Ubuntu, 7.0.0-28-generic

The exact offline vLLM 0.25.1 PoC can hard-lock the entire host without Xid,  
NVRM error, AER, MCE, watchdog output, or recoverable SSH/network access.  
Only an AC power cycle recovers it.

The PoC needs no model-weight download. It uses synthetic configs and dummy  
weights, then stops after vLLM engine construction / CUDA-graph capture.

The failure is clock-policy-sensitive:

- default high-clock policy: reproducibly wedges
- `nvidia-smi -lgc 180,2800`: clean 6/6
- `nvidia-smi -pl 600` plus `-lgc 180,2800`: clean 6/6
- `nvidia-smi -pl 350` alone: wedges, so this is not merely a power-draw issue  
The issue is reproducable in \< 6 tries

Attached MINIMAL\_POC\_sm120\_hostwedge\_one\_file.py is a standalone offline reproducer. It creates synthetic Laguna/DFlash configuration and dummy weights, then stops after LLM engine construction.

It requires no model download, model weights, tokenizer, prompt, generation, or prebuilt cache.  
The failure is a complete host lock during vLLM CUDA-graph initialization. There is no surviving Xid, NVRM error, PCIe AER, MCE, watchdog report, or network access. Only AC power removal recovers the host.

The reproducer uses vLLM 0.25.1 / FlashInfer 0.6.13, spec\_tokens=9, gpu\_memory\_utilization=0.87, and a persistent vLLM cache across attempts.

---

<div class="post-metadata">

**Author:** ![gpersoon](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/gpersoon/32/497240_2.png) [@gpersoon](https://forums.developer.nvidia.com/u/gpersoon)\
**Post date:** [August 5, 2026, 11:39am UTC](https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239/2 "2026-08-05T11:39:08Z")

</div>

SM120\_CACHE=“$PWD/sm120-poc-cache”  
mkdir -p “$SM120\_CACHE”

docker run --rm --gpus all --ipc host \  
–env HF\_HUB\_OFFLINE=1 \  
–env CUTE\_DSL\_ARCH=sm\_120a \  
–env VLLM\_USE\_FLASHINFER\_SAMPLER=0 \  
–env VLLM\_ALLOW\_LONG\_MAX\_MODEL\_LEN=1 \  
–env PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True \  
–mount type=bind,src=“$SM120\_CACHE”,dst=/root/.cache/vllm \  
–mount type=bind,src=“$PWD/MINIMAL\_POC\_sm120\_hostwedge\_one\_file.py”,dst=/poc.py,readonly \  
–entrypoint python3 vllm/vllm-openai:v0.25.1 /poc.py

---

<div class="post-metadata">

**Author:** ![gpersoon](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/gpersoon/32/497240_2.png) [@gpersoon](https://forums.developer.nvidia.com/u/gpersoon)\
**Post date:** [August 5, 2026, 11:39am UTC](https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239/3 "2026-08-05T11:39:25Z")

</div>

#!/usr/bin/env python3  
“”“Offline engine-initialization reproducer. Run twice with one cache mount.”“”

import json  
from pathlib import Path

def write\_synthetic\_configs(root: Path) → tuple[str, str]:  
target = {  
“architectures”: [“LagunaForCausalLM”],  
“model\_type”: “laguna”,  
“hidden\_size”: 3072,  
“intermediate\_size”: 12288,  
“num\_hidden\_layers”: 48,  
“max\_position\_embeddings”: 262144,  
“num\_experts”: 256,  
“num\_experts\_per\_tok”: 10,  
“moe\_intermediate\_size”: 1024,  
“shared\_expert\_intermediate\_size”: 1024,  
“torch\_dtype”: “bfloat16”,  
“gating”: “per-head”,  
“sliding\_window”: 512,  
“layer\_types”: [“full\_attention”, \*[“sliding\_attention”] \* 3] \* 12,  
“num\_attention\_heads\_per\_layer”: [48, 72, 72, 72] \* 12,  
“moe\_routed\_scaling\_factor”: 2.5,  
“rope\_parameters”: {  
“full\_attention”: {  
“rope\_theta”: 500000.0,  
“rope\_type”: “yarn”,  
“factor”: 32.0,  
“original\_max\_position\_embeddings”: 8192,  
“beta\_slow”: 1.0,  
“beta\_fast”: 32.0,  
“attention\_factor”: 1.3465735902799727,  
“partial\_rotary\_factor”: 0.5,  
},  
“sliding\_attention”: {  
“rope\_type”: “default”,  
“rope\_theta”: 10000.0,  
“partial\_rotary\_factor”: 1.0,  
},  
},  
“quantization\_config”: {  
“config\_groups”: {  
“group\_0”: {  
“format”: “nvfp4-pack-quantized”,  
“input\_activations”: {  
“actorder”: None,  
“block\_structure”: None,  
“dynamic”: “local”,  
“group\_size”: 16,  
“num\_bits”: 4,  
“observer”: “minmax”,  
“observer\_kwargs”: {},  
“scale\_dtype”: “torch.float8\_e4m3fn”,  
“strategy”: “tensor\_group”,  
“symmetric”: True,  
“type”: “float”,  
“zp\_dtype”: None,  
},  
“output\_activations”: None,  
“targets”: [  
r"re:.\*experts\.[0-9]+\.(gate\_proj|up\_proj|down\_proj)$"  
],  
“weights”: {  
“actorder”: None,  
“block\_structure”: None,  
“dynamic”: False,  
“group\_size”: 16,  
“num\_bits”: 4,  
“observer”: “memoryless\_minmax”,  
“observer\_kwargs”: {},  
“scale\_dtype”: “torch.float8\_e4m3fn”,  
“strategy”: “tensor\_group”,  
“symmetric”: True,  
“type”: “float”,  
“zp\_dtype”: None,  
},  
}  
},  
“format”: “nvfp4-pack-quantized”,  
“ignore”: [  
“lm\_head”,  
“model.layers.0.mlp.gate\_proj”,  
“model.layers.0.mlp.up\_proj”,  
“model.layers.0.mlp.down\_proj”,  
r"re:.\*\.self\_attn\.(q\_proj|k\_proj|v\_proj|o\_proj|g\_proj)$“,  
r"re:.\*\.mlp\.gate$”,  
r"re:.\*\.mlp\.shared\_expert\.(gate\_proj|up\_proj|down\_proj)$",  
],  
“kv\_cache\_scheme”: {  
“actorder”: None,  
“block\_structure”: None,  
“dynamic”: False,  
“group\_size”: None,  
“num\_bits”: 8,  
“observer”: “minmax”,  
“observer\_kwargs”: {},  
“scale\_dtype”: None,  
“strategy”: “tensor”,  
“symmetric”: True,  
“type”: “float”,  
“zp\_dtype”: None,  
},  
“quant\_method”: “compressed-tensors”,  
“quantization\_status”: “compressed”,  
},  
}  
drafter = {  
“architectures”: [“DFlashLagunaForCausalLM”],  
“model\_type”: “laguna”,  
“draft\_vocab\_size”: 100352,  
“hidden\_size”: 3072,  
“intermediate\_size”: 12288,  
“num\_hidden\_layers”: 6,  
“num\_attention\_heads”: 72,  
“max\_position\_embeddings”: 262144,  
“sliding\_window”: 512,  
“layer\_types”: [“sliding\_attention”] \* 6,  
“rope\_theta”: 10000.0,  
“gating”: “per-head”,  
“num\_experts”: 0,  
“torch\_dtype”: “bfloat16”,  
“eagle\_aux\_hidden\_state\_layer\_ids”: [2, 11, 20, 30, 39, 48],  
“dflash\_config”: {  
“block\_size”: 16,  
“mask\_token\_id”: 12,  
“num\_target\_layers”: 48,  
“target\_layer\_ids”: [1, 10, 19, 29, 38, 47],  
“causal”: True,  
},  
}

target\_dir, drafter\_dir = root / “target”, root / “drafter”  
target\_dir.mkdir(parents=True, exist\_ok=True)  
drafter\_dir.mkdir(parents=True, exist\_ok=True)  
(target\_dir / “config.json”).write\_text(json.dumps(target))  
(drafter\_dir / “config.json”).write\_text(json.dumps(drafter))  
return str(target\_dir), str(drafter\_dir)

if \ __name_\_ == “\__ main_\_”:  
from vllm import LLM

target, drafter = write\_synthetic\_configs(Path(“/tmp/sm120-wedge-poc”))  
LLM(model=target, spec\_method=“dflash”, spec\_model=drafter, spec\_tokens=9,  
max\_num\_batched\_tokens=8192, max\_num\_seqs=32, max\_model\_len=262144,  
gpu\_memory\_utilization=0.87, enable\_prefix\_caching=True, load\_format=“dummy”,  
skip\_tokenizer\_init=True)

---

<div class="post-metadata">

**Author:** ![gpersoon](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/gpersoon/32/497240_2.png) [@gpersoon](https://forums.developer.nvidia.com/u/gpersoon)\
**Post date:** [August 5, 2026, 11:40am UTC](https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239/4 "2026-08-05T11:40:35Z")

</div>

| change | observed result |
| --- | --- |
| vLLM 0.26.0 / FlashInfer 0.6.14 at utilization 0.87 | 6 clean starts; insufficient data to call fixed |
| lower speculative tokens: `spec_tokens <= 8` | 4 clean starts per tested value (1, 2, and 8) |
| graphics-clock ceiling: `sudo nvidia-smi -lgc 180,2800` | 6 clean starts at the normal 442 W limit, and 6 clean starts again with a 600 W limit; measured active graphics clocks were about 2752–2790 MHz |
| `CUDA_LAUNCH_BLOCKING=1` | 6 clean starts with the fixed `spec_tokens=9` PoC; serializes CUDA launches and substantially reduces performance |
| synchronize after every eager CUDA-graph wrapper call | 6 clean starts with the fixed PoC; requires source instrumentation and is too expensive for normal use |
| disable CUDA graphs / use eager execution | avoids the capture path, but changes runtime behavior |
| use a non-FlashInfer MoE backend (e.g. CUTLASS) | historical clean control; changes the kernel path |

---

<div class="post-metadata">

**Author:** ![njuffa](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@njuffa](https://forums.developer.nvidia.com/u/njuffa)\
**Post date:** [August 6, 2026, 5:28am UTC](https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239/5 "2026-08-06T05:28:43Z")

</div>

I am curious: Did you try asking AI for help? If so, what did it identify as the most likely root cause?

From reading the supplied information, nothing points to anything specific in my mind. It is not even clear that the hang occurs somewhere in the NVIDIA software stack.

If this were my system, I would instrument the software to (1) first find the immediate cause of the hangs (2) work backwards from there to identify the root cause. In complicated software stacks that approach might take a day or two.

---

<div class="post-metadata">

**Author:** ![gpersoon](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/gpersoon/32/497240_2.png) [@gpersoon](https://forums.developer.nvidia.com/u/gpersoon)\
**Post date:** [August 6, 2026, 6:05pm UTC](https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239/6 "2026-08-06T18:05:29Z")

</div>

Hi njuffa, thanks for thinking along.

I’ve used several AI already and they helped narrow it down. It only crashes with very specific input and specific software versions and specific frequency. So it might same kind of race condition.

This is what the AI says: Trigger (GPU, cold-calibration power-management race): On a cold-booted GB202, the PMU/GSP’s voltage-frequency calibration state is at its initial, untrained values. In that state, the top boost bins (≥2805 MHz) are marginal. The fault fires when a burst of driver-side work hits the GPU exactly during a power-state/clock transition in those bins — and vLLM startup contains one worst-case instance of this: the entry into the main CUDA-graph capture sequence, right after the FlashInfer autotune storm and KV-cache allocation. A firmware/hardware race in that V/F transition freezes a GPU domain while its PCIe interface still holds in-flight protocol state — credits not released, accepted transactions never completed.

A timeline of a capture of the issue:  
16:39:54 — Container starts, tokenizer loaded  
16:39:58 — vLLM engine initialization begins  
16:39:59 — Model load starts; FlashInfer NVFP4 MoE backend selected  
16:40:00 — Target model checkpoint identified: 66.98 GiB  
16:40:45 — Target model weights finished loading (45.2s)  
16:40:45 — Drafter (DFlash) model load starts  
16:40:46 — Drafter weights finished loading (1.1s)  
16:40:47 — Full model load complete: 69.26 GiB, 47.0s total  
16:40:52–16:41:12 — Target model compiled via torch.compile (24.7s)  
16:41:14–16:41:19 — Drafter model compiled (4.7s)  
16:41:23 — Graph mode set to PIECEWISE; profiling begins for 51 graph sizes (largest 512 tokens)  
16:41:24 — KV cache sized: 9.81 GiB, 268,024 tokens  
16:41:25 — FlashInfer autotuning starts  
16:41:36 — Autotuning ends, 42 configs saved  
16:41:36 — Attention warmup runs  
16:41:37 — “Skipping CuTeDSL warmup” — last line printed  
16:41:39 — Host stops responding to network requests  
16:41:39–16:42:51 — Kernel heartbeat continues normally, then stops — no partial beat, no slowdown

---

<div class="post-metadata">

**Author:** ![njuffa](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@njuffa](https://forums.developer.nvidia.com/u/njuffa)\
**Post date:** [August 6, 2026, 6:54pm UTC](https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-ws-whole-host-hard-lock-during-vllm-cuda-graph/379239/7 "2026-08-06T18:54:46Z")

</div>

> [@gpersoon](#):
>
> A firmware/hardware race in that V/F transition freezes a GPU domain while its PCIe interface still holds in-flight protocol state — credits not released, accepted transactions never completed.

My understanding of PCIe does not go down to that level of detail, but my expectation would be that losing communication with a GPU (due to the GPU’s PCIe interface getting wedged) would not cause the entire operating system to halt.

This is the problem with AI output: It often _sounds_ eminently plausible, especially where it exceeds one’s own knowledge, but it is impossible to tell whether it might not be hallucinating. As for the potential root cause identified, a race condition during a power-state transition of the GPU: It seems plausible that these occur, but one would also expect GPU power management to guard against it, for example by waiting until PCIe interface is idle before switching the state.

If in fact the GPU power management is to blame, this is an issue that would need to be resolved by NVIDIA engineering, i.e. you would need to file a bug with NVIDIA. There used to be a sticky link at the top of the forum providing instructions on how to do that.