Everyone seems to have settled into two camps with this model: SGLang on two Sparks, or vLLM on one. Nobody had tried the obvious third thing — vLLM across four Sparks at TP4 — so I did. It’s up, it’s gated, and the repo has everything:
GitHub - tsw2k/Qwen3.8-Flash-Next-Quad-DGX-Sparks · GitHub
The headline numbers (TP4, native 262k, MTP=2, thinking off)
| KV pool (bf16) | 4,696,556 tokens — 17.92x a full 262k request |
|---|---|
| Single-stream decode | 31.0 tok/s — the fabric costs nothing (TP1 baseline on the same image: 30.8) |
| Aggregate decode | 46 (x2) · 53 (x4) · 97 (x8) · 157 tok/s (x16) |
| TTFT | 0.24 s (x1) → 1.25 s (x16) |
| Boot | ~8 min — sharded weight load actually beats single-node’s 14.5 |
| Gate suite | all PASS — concurrent ~30k prefills, byte-identical greedy decode, NIAH 4k/32k/128k at 0/50/100% depth |
| MTP acceptance | 0.856 on mixed traffic |
| Also verified | vision input, tool calling (clean tool_calls), prefix caching (0.7 s TTFT on a cached 20k prefix), thinking |
The memory story in one line: one Spark fits 1.73 full contexts of KV next to this model; four fit 17.92. vLLM’s QSA layers refuse anything but bf16 KV (~30 GiB per 1M-token YaRN request), which is why the single-box vLLM recipe caps at 500k. Weight sharding is what buys the room back.
The 1M rung: half-conquered, honestly reported
The 1M YaRN lane boots and serves: a 4,996k-token KV pool — 4.78x a full 1,048,576-token request — and NIAH-128k passes at every depth. Then the interesting part: the ~3rd request deeper than ~100k tokens wedges a GPU stream and takes the engine with it. I’ve eliminated prefix caching, MTP, CUDA graphs (--enforce-eager dies identically), allocator settings, page cache, chunk size, and one very educational systemd unit — one variable per boot, on a rebooted fleet each time. The captured py-spy stack and the full isolation matrix are in the repo’s docs/OPEN-PROBLEMS.md; one suspect remains before this goes upstream as a kernel bug. So: the 262k lane above is the shipped default, the 1M lane is experimental — a config that answers one 1M-deep prompt is not a config that works.
What actually happened when I hit “launch”
Boot #1 died in seconds. Boot #2 died differently — which on GB10 counts as progress. Three fixes later it serves, and none of them appear in any single- or dual-node recipe (all are launcher defaults in the repo now):
- Plain TP4 cannot load this checkpoint. At all. TP sharding slices the MoE intermediate 640 → 160 per rank, and the NVFP4 FLASHINFER_CUTLASS backend answers with
NotImplementedError: Intermediate size padding for w1 and w3. The cure is--enable-expert-parallel: don’t slice the experts, deal them out whole — 512/4 = 128 per rank. - Your fd limit is about to overflow and you don’t know it yet. The PLE n-gram table is 128 memmapped safetensors shards. Add the socket storm of 4-node NCCL + EP on top and Docker’s default
nofilefolds —OSError: [Errno 24] Too many open files, ten minutes into the boot, deep inside PLE setup.--ulimit nofile=1048576and it never bothers you again. - Do not hardcode
NCCL_IB_GID_INDEX. After a link bounce, the RoCE v2 GID can land on a different index on one node (I caught index 4 vs 3 fleet-wide). A pinned index then reads a zero GID and NCCL greets you withunhandled system errorat init. WithNCCL_IB_ROCE_VERSION_NUM=2andNCCL_IB_ADDR_RANGEset, NCCL picks the right GID per device by itself.
Free ops lesson, learned the slow way
Do not move a 122 GiB checkpoint between Sparks over ssh. One AES stream pins a GB10 ARM core at ~1 Gbps and your afternoon is gone. A read-only rsync daemon on the compute rails does it at ~16 Gbps — ~2 minutes per node. Script’s in the repo, along with the image fan-out that verifies the docker image ID (not the tag) on every node before any launch.
What’s in the repo
A Dockerfile over the official day-0 vLLM image with the known GB10 patch stack vendored in, a 4-node worker-first launcher with the full NCCL env, fleet scripts (weight/image fan-out, preflight, teardown, self-healing watchdog), the gate suite, and the bench harness that produced every number above. Standing on a lot of shoulders: blazux’s single-Spark vLLM recipe (with work by Saren-Arterius, jschmied, k3dani — full NOTICE in the repo), tonyd2wild’s 4-Spark GLM ops discipline, and MiaAI-Lab’s dual-Spark SGLang recipe that mapped this model’s memory cliffs first.
Next
Closing the 1M wedge (one suspect left, then it goes upstream with the stack), the KV ladder, the fp8-hybrid side layers at TP4 — numbers land in the README as they’re measured. One client gotcha meanwhile: this vLLM build returns chain-of-thought in message.reasoning, not reasoning_content — wire your agent accordingly.
If anyone else is running vLLM multi-node on Sparks — especially expert parallelism over RoCE — comments very welcome. I have a switch, four nodes, and now also a working endpoint.