The Issue (TL;DR)
When serving the dense Qwen/Qwen3.8-27B-FP8 via vLLM on a DGX Spark, the engine silently crashes on the first large agentic request (~6.5K tokens). The container exits with code 255, and 2–3.5 minutes later, the entire DGX Spark machine hard-resets. This reproduces 100% of the time.
Environment
-
Hardware: NVIDIA DGX Spark, GB10, Compute Capability
sm_121, 119.7 GiB UMA pool. -
Software: Driver 580.95.05, Kernel 6.14.0-1013-nvidia.
-
Engine:
vllm/vllm-openai:cu130-nightly
The Exact Failure Signature
This does not behave like a standard CUDA OOM:
-
Engine serves normally until the first large prefill (cold start).
-
vLLM log stream abruptly stops mid-request. No Python traceback, no CUDA error, no Xid in
dmesg. -
Container exits
255. -
Docker reports
OOMKilled=falseand kerneloom_killcounter is 0. -
Host telemetry showed ~50 GiB MemAvailable and low KV-cache (~5%) right before the crash.
-
2–3.5 minutes later, the whole machine hard-resets (power-cut signature).
What I’ve Tried
-
Context Cap: Lowered
max_model_lento 32768 (still fails/resets). -
Eager Mode: Set
enforce_eager: trueto disable CUDA graphs and free ~28GiB. (still fails/resets). -
The Control: Dropped the dense model and swapped to
Qwen3.6-35B-A3B(MoE) on the exact same stack and tool-calling workload. It has been 100% stable. The crash seems tied specifically to the dense model’s active working set or kernel path.
Questions
-
Can a transient CUDA/vLLM allocation spike on GB10 unified memory wedge the driver so severely that it triggers a delayed hardware reset without logging an ordinary OOM/Xid?
-
Are there known kernel crashes for dense FP8 on GB10 (SM121) with
580.95.05?