Total host freeze (not process hang) during multi-node TP=2 vLLM prefill on 2× DGX Spark GB10, zero forensic trace across kdump/watchdogs/netconsole

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Disclaimer: LLM usage

Config: 2x DGX Spark (GB10), TP=2 vLLM cluster via `launch-cluster.sh`+Ray, serving Step-3.7-Flash-NVFP4 via: GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub .

During long/resumed-chat prompts (heavy non-cached prefill), one node totally freezes — no ping, no SSH, no display — 5 times across 2 days. This does not occur on normal chat though, even as context fills (70% of 256k is when harness compresses the chat history).

An initial unified-memory OOM bug (percentage-based `gpu_memory_utilization` with no floor) was found and fixed, but the host-freezes continued afterward with that fix confirmed active. Every freeze leaves zero forensic trace: no OOM-killer, no kernel panic, no NVRM/Xid GPU fault, no hung-task/softlockup warning, and kdump never once produces a vmcore despite being enabled.

I added kernel watchdog panics (`hung_task_panic`, `softlockup_panic`), bidirectional `netconsole`, and PyTorch’s NCCL Flight Recorder to try to catch it — still nothing, even the disk-independent network-based logging never transmits anything before the freeze.

This suggests interrupts/scheduler/NIC all wedge simultaneously, pointing toward a hardware/firmware-level lockup rather than a pure software hang.

Driver is 580.159.03, kernel 6.17.0-1026-nvidia, CUDA 13.0.

Looking for anyone who’s hit the same “total host death during heavy multi-node prefill” pattern (not just a stuck inference process) or ideas for what else could capture forensic evidence of this.

I’ve attached a LLM generated report of what was tried and a bug-report archive from each spark.

Thank you!

forum-full-report.md.zip (9.3 KB)

nvidia-bug-report-spark.log.gz (505.1 KB)

nvidia-bug-report-spark2.log.gz (696.1 KB)

2

Thermal shutdown. Plenty of posts for you to sift through. Quick list of techniques thus far: 3DP fan attachments, underclock, external fans, heatsinks, remove the case, repaste, RMA if fail field test. There was slight talk of custom water cooling but that hasn’t surfaced yet.

3

The units are in air-conditioned rooms with 120mm 4k rpm fans blowing over them, so I didn’t even think thermal would be a problem. but I ran the field test and it failed, so time for an RMA.

4

Sorry you’re hitting this — we just came out the other side of a multi-day freeze hunt on a 4× DGX Spark (GB10) cluster serving a large model with vLLM, and two platform issues we pinned down might be relevant to yours.

  1. A GPU kernel livelock that presents as a frozen box. The flashinfer sparse_mla_sm120 attention kernels can livelock on GB10 (mbarrier arrive/wait race; cuda-gdb receipt in the links below). Tell-tale signature: nvidia-smi shows ~96 % GPU utilization but only ~18–21 W power and ~0 % memory throughput — the GPU is spinning, not computing, and host processes stack up behind it in cuLaunchKernel. If your workload is a sparse-MLA model (DeepSeek/GLM lineage), the fix for us was one flag — --attention-backend FLASHMLA_SPARSE — plus a small set of drop-in Triton attention files for sm12x (recipe in the links). Zero freezes since, across 500+ consecutive long-context sessions.
  2. A driver memory-descriptor leak worth ruling out. Check journalctl -k | grep -c 0x00000051 — GB10’s unified memory driver (NVRM 580.159.03 and earlier) leaks memdescs under allocation churn, and a box that has been serving for a while can start failing allocations and grinding into swap. A reboot resets it; idle time does not spend it, allocation churn does.

Our full write-up with receipts is in the vllm-project/vllm RFC #48720 results update ([RFC] TP-wide step-consensus guard with NCCL RAS integration — field-validated, two live deadlock captures · Issue #48720 · vllm-project/vllm · GitHub), and the sanitized logs/stack captures are in the evidence repo (GitHub - marksunner/glm52-dgx-spark-deadlock-evidence: GLM-5.2 on 4x DGX Spark: FlashInfer sparse-MLA mbarrier livelock - root-cause evidence, validated workaround, and reproducer pack · GitHub). If your freeze doesn’t match either signature, the cheapest discriminators we found were nvidia-smi power draw during the hang and NCCL RAS per-rank op counts (nc localhost 28028).

5

Thanks for the reply. I exchanged both of my units (both were from right at release) through micro center. I immediately field tested the two new units and both passed. so then I spun up spark-vllm-docker; had my llm update a patch to force vllm to not use more then 90gb of memory, and haven’t had issues for the last couple days.

So why is that important: I’m pretty sure the vllm wasn’t the issue, but rather the units. Why: in my own testing the thermal throttling was inconsistent in when and how it came into effect, by watching the clock speed.

Hopefully the fixes you’re bringing just make things better for me, thank you for the hard work!