Up front, a fair concession. If you want raw aggregate throughput, the config
in this post (tp=4, ep=4) isn’t how you’d get it — splitting one model across four
nodes pays an all-reduce tax every layer. The axis that actually scales is
--dp-size 4 without --enable-dp-attention: four full replicas behind SGLang’s
router, zero cross-node comms, ~4× the single-node rate, and the model fits on one
128 GB Spark so replication works. We haven’t benched that path yet (--dp-size + MTP
is untested for us), so it’s the honest “what we’d reach for next,” not a measured claim.
So why TP=4 at all?
Because the point of the post isn’t the topology — it’s that MTP works, which it
didn’t before on any node count. The published state was:
- “crashes” — vLLM, t/366660
- “0 % acceptance, accept_len = 1.00” — SGLang, sglang#21138
…single-node attempts included. On a build with the June-2026 NemotronH-MTP fixes it
pays off: accept_len ≈ 2.7, 1.70× single-stream, and 3/4 beating NVIDIA’s own
cookbook 5/5. That’s the result worth sharing. TP=4 was just the bench the cluster
already serves one model on, and the recipe holds regardless of how you parallelize.
“Why not one Spark, then?” — we actually tried it
This is the interesting part. The weights are only ~67 GB, so they fit on a single
128 GB box. But on this hybrid the concurrency limit isn’t KV — it’s the Mamba state
pool:
max_running_requests = max_mamba_cache_size // per-request-slots
On one Spark the full model loads unsharded to ~72.7 GB, leaving just ~37.5 GB
free. The Mamba pool we run on the cluster (96 slots) wants ~27 GB on top of KV +
CUDA graphs — which doesn’t fit:
RuntimeError: Not enough memory. Please try to increase --mem-fraction-static.
We did get it running single-node, but only after two cuts (and the engine’s own
“increase --mem-fraction-static” advice is a red herring — avail mem after weights
is physically pinned at ~37.5 GB regardless of the fraction):
- shrink the Mamba pool (
max_mamba_cache_size96 → 24) to clear the pool-init OOM, then - lower
mem-fraction-static(≈0.78) so the KV pool stops eating the headroom the
MTP draft head (~7 GB) needs — raising it is the wrong reflex in both phases.
Measured single-node memory budget (one Spark, ~121 GB usable, MTP on):
| Item | GB |
|---|---|
| Base weights (unsharded) | 72.7 |
| Mamba pool (24 slots) | 7.1 |
| KV cache (885k tokens, fp8) | 3.4 |
| MTP draft head | 7.0 |
| Draft KV + CUDA graphs | ~1.5 |
| free at the end | ~16 |
The result boots and serves — MTP works on one Spark too (accept_len ≈ 2.4) — but the
concurrency ceiling is max_running_requests = 4, gated by the Mamba state pool, vs
the cluster’s ~32 (≈19 with MTP). Under TP=4 that pool is sharded across nodes (~17 GB
weights/node), so it fits at full size.
And there’s a second ceiling the KV cut introduces: with the pool trimmed to ~885k
tokens, those 4 slots share a single KV budget — ~221k tokens each if all four run at
once. So a single Spark can serve either 4 concurrent at moderate context, or fewer at
the full 512 K — not four long-context requests at the same time. Sharding across nodes
restores both ceilings: more Mamba slots (parallelism) and more aggregate KV (long
context × concurrency together) — which is the whole reason the model’s 512 K context is
worth anything in a multi-user setting.
The honest shape of it
Four Sparks aren’t required — they buy more usable concurrency × context by
sharding the Mamba and KV pools, not single-stream magic.
If we’re being fully honest, two Sparks would probably be the sweet spot:
- TP=2 halves the all-reduce of TP=4
- restores a healthy Mamba pool (~36 GB weights/node, tons of headroom)
- KV pool no longer has to be starved to fit the MTP head
- model still fits with room to spare
We just happen to have four, so four is what got benched.
The KV-cache angle is backwards here
The reason people usually cite for going multi-node — distributing a growing KV cache —
doesn’t apply. NemotronH is a hybrid: only 8 of 88 layers are attention (the other 80
are Mamba, carrying a fixed recurrent state), so KV barely grows (~4 KB/token) and 512 K
context is nearly free per request. The thing actually worth distributing is the Mamba
state pool — and, as the single-node run shows, the aggregate KV budget once you want
long context across several requests at once.
So, to actually answer the question honestly: if you only ever do single requests, one
Spark is completely sufficient. It boots, it serves, MTP works, and a lone request gets
the full 512 K context — the second node earns nothing for you. Four Sparks only start to
pay off the moment you want many requests in flight and long context at the same time:
that’s when the sharded Mamba + KV pools turn into real, usable headroom.
We have four, so the cluster just runs it — clean 8/8, MTP on, no babysitting. But
that was never the headline. The headline is that MTP works; the node count is just
where you land on the concurrency-vs-cost curve. One Spark for yourself, two for a
comfortable sweet spot, four if you’re serving a crowd.