DGX Spark GB10 hard-resets after vLLM silently exits 255 serving Qwen3.8-27B-FP8 — no OOM, Xid, or traceback

The Issue (TL;DR)
When serving the dense Qwen/Qwen3.8-27B-FP8 via vLLM on a DGX Spark, the engine silently crashes on the first large agentic request (~6.5K tokens). The container exits with code 255, and 2–3.5 minutes later, the entire DGX Spark machine hard-resets. This reproduces 100% of the time.

Environment

  • Hardware: NVIDIA DGX Spark, GB10, Compute Capability sm_121, 119.7 GiB UMA pool.

  • Software: Driver 580.95.05, Kernel 6.14.0-1013-nvidia.

  • Engine: vllm/vllm-openai:cu130-nightly

The Exact Failure Signature
This does not behave like a standard CUDA OOM:

  1. Engine serves normally until the first large prefill (cold start).

  2. vLLM log stream abruptly stops mid-request. No Python traceback, no CUDA error, no Xid in dmesg.

  3. Container exits 255.

  4. Docker reports OOMKilled=false and kernel oom_kill counter is 0.

  5. Host telemetry showed ~50 GiB MemAvailable and low KV-cache (~5%) right before the crash.

  6. 2–3.5 minutes later, the whole machine hard-resets (power-cut signature).

What I’ve Tried

  • Context Cap: Lowered max_model_len to 32768 (still fails/resets).

  • Eager Mode: Set enforce_eager: true to disable CUDA graphs and free ~28GiB. (still fails/resets).

  • The Control: Dropped the dense model and swapped to Qwen3.6-35B-A3B (MoE) on the exact same stack and tool-calling workload. It has been 100% stable. The crash seems tied specifically to the dense model’s active working set or kernel path.

Questions

  1. Can a transient CUDA/vLLM allocation spike on GB10 unified memory wedge the driver so severely that it triggers a delayed hardware reset without logging an ordinary OOM/Xid?

  2. Are there known kernel crashes for dense FP8 on GB10 (SM121) with 580.95.05?

What backend do you use? (it’s recommended to use flashinfer but normally it’s the default choice if not specified)

Try with spark-vllm-docker it works for me.

Thanks. Unfortunately, my issue is a defective thermostat

https://forums.developer.nvidia.com/t/re-dgx-spark-gb10-hard-resets-after-vllm-silently-exits-255-serving-qwen3-8-27b-fp8-no-oom-xid-or-traceback/380302

The diagnostics resulting in a failed message.

@jonathan.lugo You are on quite an old software stack. Please update your system to the latest version through the DGX Dashboard and try to reproduce

I am confused by your message. I already got an RMA and Nvidia (you) will repair my DGX. Or you meant something else? Many thanks.

Good to hear that, thanks for letting me know. No further action needs to be taken then.