Two models on one DGX Spark: Qwen3.6-35B NVFP4 + Nemotron 3.5 Lightning with DSpark, numbers under shared load and a fix for cuDNN FP8 crashes

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, ,
1

We have been running Qwen3.6-35B-A3B (NVIDIA’s NVFP4 checkpoint) and Nemotron 3.5 Lightning (NVFP4 + DSpark draft) side by side on one DGX Spark in production since September, vLLM 0.28.1 (timothystewart6/vllm-gb10 image). A few findings that might save others some time:

  1. NVFP4 was faster for us, not slower. Qwen3.6-35B single stream: 52.6 tok/s (FP8, vLLM 0.18) vs 77.9 tok/s (NVIDIA’s mixed NVFP4/FP8 checkpoint, vLLM 0.28). Neither change alone was enough: vLLM 0.28 with the FP8 checkpoint was slower than 0.18.

  2. Two models share one memory bus. At 8 concurrent requests, each alone: Qwen 263.7, Nemotron 293.5 tok/s. Both under the same load at once: 107.7 and 149.6, so together about 257, less than either one alone. Two models on one Spark buy flexibility, not throughput. TTFT roughly doubles.

  3. Random crashes with 1 to 3 concurrent requests after switching to NVFP4 (EngineDeadError). In our case the cause was FlashInfer #3566: vLLM calls the FP8 GEMM with backend=“auto”, which picks cuDNN on GB10, and that kernel fails occasionally at small batch sizes. Forcing “cublas” in vllm/utils/flashinfer.py explained 7 of our 9 crashes, with no throughput loss. No crash since the patch, but we will only call it solved after 72 days.

  4. vLLM exits with code 0 after EngineDeadError, so systemd Restart=on-failure never fires. Use Restart=always.

  5. In vLLM 0.28 prefix caching is on by default. Omitting the flag does nothing; you need --no-enable-prefix-caching.

  6. DSpark gains depend heavily on the text: 55 % acceptance and 112 tok/s on short prompts, but only 37 to 40 % on real 32k-token input, where Nemotron drops to about 85 tok/s, barely above its no-draft value of 79. A synthetic repeated-sentence test had shown 117 to 157 tok/s, which was misleading.

  7. Memory watchdog for first starts: check /proc/meminfo every second, docker kill when MemAvailable < 12,000 MiB or swap grows by more than 8,000 MiB. docker stats showed 7.7 GiB for a container that nvidia-smi showed at 52.9 GiB.

  8. On our two-node setup, decode dropped from 48 to 34 tok/s after ~43 days of uptime: kcompactd0 at ~100 % trying to move GPU-pinned pages. vm.compaction_proactiveness = 0 fixed it.

Full write-up with configs, the patch script, benchmark script and method, memory breakdown and everything that went wrong:

Happy to answer questions or compare numbers.

1 Like