RTX PRO 6000 Blackwell WS: whole-host hard lock during vLLM CUDA-graph

Accelerated Computing CUDA CUDA Programming and Performance
1

My entire PC was crashing on a specific vLLM load - i was able to reproduce it, see below.
Can anyone else reproduce it ? (if not it might be related to my setup)
Also reported here: Log in | NVIDIA Developer
Details:
GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
VBIOS: 98.02.81.00.07
Driver: 610.43.03 open kernel module, CUDA 13.3
Host: ASUS ROG STRIX X870E-E, Ryzen 9 9950X3D, 64 GiB DDR5-4800
OS/kernel: Ubuntu, 7.0.0-28-generic

The exact offline vLLM 0.25.1 PoC can hard-lock the entire host without Xid,
NVRM error, AER, MCE, watchdog output, or recoverable SSH/network access.
Only an AC power cycle recovers it.

The PoC needs no model-weight download. It uses synthetic configs and dummy
weights, then stops after vLLM engine construction / CUDA-graph capture.

The failure is clock-policy-sensitive:

  • default high-clock policy: reproducibly wedges
  • `nvidia-smi -lgc 180,2800`: clean 6/6
  • `nvidia-smi -pl 600` plus `-lgc 180,2800`: clean 6/6
  • `nvidia-smi -pl 350` alone: wedges, so this is not merely a power-draw issue
    The issue is reproducable in < 6 tries

Attached MINIMAL_POC_sm120_hostwedge_one_file.py is a standalone offline reproducer. It creates synthetic Laguna/DFlash configuration and dummy weights, then stops after LLM engine construction.

It requires no model download, model weights, tokenizer, prompt, generation, or prebuilt cache.
The failure is a complete host lock during vLLM CUDA-graph initialization. There is no surviving Xid, NVRM error, PCIe AER, MCE, watchdog report, or network access. Only AC power removal recovers the host.

The reproducer uses vLLM 0.25.1 / FlashInfer 0.6.13, spec_tokens=9, gpu_memory_utilization=0.87, and a persistent vLLM cache across attempts.

2

SM120_CACHE=“$PWD/sm120-poc-cache”
mkdir -p “$SM120_CACHE”

docker run --rm --gpus all --ipc host \
–env HF_HUB_OFFLINE=1 \
–env CUTE_DSL_ARCH=sm_120a \
–env VLLM_USE_FLASHINFER_SAMPLER=0 \
–env VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
–env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
–mount type=bind,src=“$SM120_CACHE”,dst=/root/.cache/vllm \
–mount type=bind,src=“$PWD/MINIMAL_POC_sm120_hostwedge_one_file.py”,dst=/poc.py,readonly \
–entrypoint python3 vllm/vllm-openai:v0.25.1 /poc.py

3

#!/usr/bin/env python3
“”“Offline engine-initialization reproducer. Run twice with one cache mount.”“”

import json
from pathlib import Path

def write_synthetic_configs(root: Path) → tuple[str, str]:
target = {
“architectures”: [“LagunaForCausalLM”],
“model_type”: “laguna”,
“hidden_size”: 3072,
“intermediate_size”: 12288,
“num_hidden_layers”: 48,
“max_position_embeddings”: 262144,
“num_experts”: 256,
“num_experts_per_tok”: 10,
“moe_intermediate_size”: 1024,
“shared_expert_intermediate_size”: 1024,
“torch_dtype”: “bfloat16”,
“gating”: “per-head”,
“sliding_window”: 512,
“layer_types”: [“full_attention”, *[“sliding_attention”] * 3] * 12,
“num_attention_heads_per_layer”: [48, 72, 72, 72] * 12,
“moe_routed_scaling_factor”: 2.5,
“rope_parameters”: {
“full_attention”: {
“rope_theta”: 500000.0,
“rope_type”: “yarn”,
“factor”: 32.0,
“original_max_position_embeddings”: 8192,
“beta_slow”: 1.0,
“beta_fast”: 32.0,
“attention_factor”: 1.3465735902799727,
“partial_rotary_factor”: 0.5,
},
“sliding_attention”: {
“rope_type”: “default”,
“rope_theta”: 10000.0,
“partial_rotary_factor”: 1.0,
},
},
“quantization_config”: {
“config_groups”: {
“group_0”: {
“format”: “nvfp4-pack-quantized”,
“input_activations”: {
“actorder”: None,
“block_structure”: None,
“dynamic”: “local”,
“group_size”: 16,
“num_bits”: 4,
“observer”: “minmax”,
“observer_kwargs”: {},
“scale_dtype”: “torch.float8_e4m3fn”,
“strategy”: “tensor_group”,
“symmetric”: True,
“type”: “float”,
“zp_dtype”: None,
},
“output_activations”: None,
“targets”: [
r"re:.*experts\.[0-9]+\.(gate_proj|up_proj|down_proj)$"
],
“weights”: {
“actorder”: None,
“block_structure”: None,
“dynamic”: False,
“group_size”: 16,
“num_bits”: 4,
“observer”: “memoryless_minmax”,
“observer_kwargs”: {},
“scale_dtype”: “torch.float8_e4m3fn”,
“strategy”: “tensor_group”,
“symmetric”: True,
“type”: “float”,
“zp_dtype”: None,
},
}
},
“format”: “nvfp4-pack-quantized”,
“ignore”: [
“lm_head”,
“model.layers.0.mlp.gate_proj”,
“model.layers.0.mlp.up_proj”,
“model.layers.0.mlp.down_proj”,
r"re:.*\.self_attn\.(q_proj|k_proj|v_proj|o_proj|g_proj)$“,
r"re:.*\.mlp\.gate$”,
r"re:.*\.mlp\.shared_expert\.(gate_proj|up_proj|down_proj)$",
],
“kv_cache_scheme”: {
“actorder”: None,
“block_structure”: None,
“dynamic”: False,
“group_size”: None,
“num_bits”: 8,
“observer”: “minmax”,
“observer_kwargs”: {},
“scale_dtype”: None,
“strategy”: “tensor”,
“symmetric”: True,
“type”: “float”,
“zp_dtype”: None,
},
“quant_method”: “compressed-tensors”,
“quantization_status”: “compressed”,
},
}
drafter = {
“architectures”: [“DFlashLagunaForCausalLM”],
“model_type”: “laguna”,
“draft_vocab_size”: 100352,
“hidden_size”: 3072,
“intermediate_size”: 12288,
“num_hidden_layers”: 6,
“num_attention_heads”: 72,
“max_position_embeddings”: 262144,
“sliding_window”: 512,
“layer_types”: [“sliding_attention”] * 6,
“rope_theta”: 10000.0,
“gating”: “per-head”,
“num_experts”: 0,
“torch_dtype”: “bfloat16”,
“eagle_aux_hidden_state_layer_ids”: [2, 11, 20, 30, 39, 48],
“dflash_config”: {
“block_size”: 16,
“mask_token_id”: 12,
“num_target_layers”: 48,
“target_layer_ids”: [1, 10, 19, 29, 38, 47],
“causal”: True,
},
}

target_dir, drafter_dir = root / “target”, root / “drafter”
target_dir.mkdir(parents=True, exist_ok=True)
drafter_dir.mkdir(parents=True, exist_ok=True)
(target_dir / “config.json”).write_text(json.dumps(target))
(drafter_dir / “config.json”).write_text(json.dumps(drafter))
return str(target_dir), str(drafter_dir)

if _name_ == “_main_”:
from vllm import LLM

target, drafter = write_synthetic_configs(Path(“/tmp/sm120-wedge-poc”))
LLM(model=target, spec_method=“dflash”, spec_model=drafter, spec_tokens=9,
max_num_batched_tokens=8192, max_num_seqs=32, max_model_len=262144,
gpu_memory_utilization=0.87, enable_prefix_caching=True, load_format=“dummy”,
skip_tokenizer_init=True)

4
change observed result
vLLM 0.26.0 / FlashInfer 0.6.14 at utilization 0.87 6 clean starts; insufficient data to call fixed
lower speculative tokens: spec_tokens <= 8 4 clean starts per tested value (1, 2, and 8)
graphics-clock ceiling: sudo nvidia-smi -lgc 180,2800 6 clean starts at the normal 442 W limit, and 6 clean starts again with a 600 W limit; measured active graphics clocks were about 2752–2790 MHz
CUDA_LAUNCH_BLOCKING=1 6 clean starts with the fixed spec_tokens=9 PoC; serializes CUDA launches and substantially reduces performance
synchronize after every eager CUDA-graph wrapper call 6 clean starts with the fixed PoC; requires source instrumentation and is too expensive for normal use
disable CUDA graphs / use eager execution avoids the capture path, but changes runtime behavior
use a non-FlashInfer MoE backend (e.g. CUTLASS) historical clean control; changes the kernel path
5

I am curious: Did you try asking AI for help? If so, what did it identify as the most likely root cause?

From reading the supplied information, nothing points to anything specific in my mind. It is not even clear that the hang occurs somewhere in the NVIDIA software stack.

If this were my system, I would instrument the software to (1) first find the immediate cause of the hangs (2) work backwards from there to identify the root cause. In complicated software stacks that approach might take a day or two.

6

Hi njuffa, thanks for thinking along.

I’ve used several AI already and they helped narrow it down. It only crashes with very specific input and specific software versions and specific frequency. So it might same kind of race condition.

This is what the AI says: Trigger (GPU, cold-calibration power-management race): On a cold-booted GB202, the PMU/GSP’s voltage-frequency calibration state is at its initial, untrained values. In that state, the top boost bins (≥2805 MHz) are marginal. The fault fires when a burst of driver-side work hits the GPU exactly during a power-state/clock transition in those bins — and vLLM startup contains one worst-case instance of this: the entry into the main CUDA-graph capture sequence, right after the FlashInfer autotune storm and KV-cache allocation. A firmware/hardware race in that V/F transition freezes a GPU domain while its PCIe interface still holds in-flight protocol state — credits not released, accepted transactions never completed.

A timeline of a capture of the issue:
16:39:54 — Container starts, tokenizer loaded
16:39:58 — vLLM engine initialization begins
16:39:59 — Model load starts; FlashInfer NVFP4 MoE backend selected
16:40:00 — Target model checkpoint identified: 66.98 GiB
16:40:45 — Target model weights finished loading (45.2s)
16:40:45 — Drafter (DFlash) model load starts
16:40:46 — Drafter weights finished loading (1.1s)
16:40:47 — Full model load complete: 69.26 GiB, 47.0s total
16:40:52–16:41:12 — Target model compiled via torch.compile (24.7s)
16:41:14–16:41:19 — Drafter model compiled (4.7s)
16:41:23 — Graph mode set to PIECEWISE; profiling begins for 51 graph sizes (largest 512 tokens)
16:41:24 — KV cache sized: 9.81 GiB, 268,024 tokens
16:41:25 — FlashInfer autotuning starts
16:41:36 — Autotuning ends, 42 configs saved
16:41:36 — Attention warmup runs
16:41:37 — “Skipping CuTeDSL warmup” — last line printed
16:41:39 — Host stops responding to network requests
16:41:39–16:42:51 — Kernel heartbeat continues normally, then stops — no partial beat, no slowdown

7

My understanding of PCIe does not go down to that level of detail, but my expectation would be that losing communication with a GPU (due to the GPU’s PCIe interface getting wedged) would not cause the entire operating system to halt.

This is the problem with AI output: It often sounds eminently plausible, especially where it exceeds one’s own knowledge, but it is impossible to tell whether it might not be hallucinating. As for the potential root cause identified, a race condition during a power-state transition of the GPU: It seems plausible that these occur, but one would also expect GPU power management to guard against it, for example by waiting until PCIe interface is idle before switching the state.

If in fact the GPU power management is to blame, this is an issue that would need to be resolved by NVIDIA engineering, i.e. you would need to file a bug with NVIDIA. There used to be a sticky link at the top of the forum providing instructions on how to do that.