Does anyone have a serving recipe for the nvidia nvfp GLM5.3 Flash model for a 4 spark cluster in TP=4? We’ve found two spark cluster recipes, but nothing for four. Thanks!
Three four-node recipes are public:
alexellis/glm-5.3-flash-4x-dgx-spark-switchless: TP4 over a switchless RoCE ring with the DFlash2 drafter, 262K context, around 45 tok/s on agentic traffic. It pinsLibertAIDAI/GLM-5.3-Flash-NVFP4at9e0d74e3.tonyd2wild/GLM-5.3-Flash-NVFP4-1M-KV-4x-DGX-Spark: TP4 at the full 1,048,576 context with a 3,834,498 token fp8 KV pool, DFlash2,--moe-backend marlin, default checkpointRedHatAI/GLM-5.3-Flash-NVFP4. The same images run at TP2 at 262K, so you can diff it against the two-node recipe you already have.mpfaffenberger/GLM-5.3-Flash-TP4-4x-DGX-Spark:unsloth/GLM-5.3-Flash-FP8, native NCCL without Ray, 36 to 37 tok/s at C=1, published default 65K with--max-num-seqs 4. It marks 100K at concurrency 10 unsupported, citing a KDA kernel wedge.
alexellis runs tonyd2wild’s image ghcr.io/tonyd2wild/vllm-glm53-flash:sm121-v11-dflash2 and says its own build is not distributed. Only mpfaffenberger publishes one of its own.
One note on the checkpoint you named, since the first recipe pins a different NVFP4 build and the second moved off that family. Both trace to vLLM #54150, invalid UTF-8 byte tokens returned at HTTP 200 with normal throughput. It was root caused to modelopt.py, where the fused gate and up GEMM takes w13_weight_scale_2[:, 0], so the up half gets dequantized with gate’s global scale whenever the two differ. nvidia/GLM-5.3-Flash-NVFP4 is ModelOpt produced and does take that path, so I range-read its safetensors headers and pulled every per-expert gate_proj.weight_scale_2 and up_proj.weight_scale_2. All 12,096 pairs are identical, across all 42 MoE layers and all 288 experts. The single gscale shortcut is a no-op on it.
Stock vLLM stops earlier than any of this, at pe_dim must be 64 for fp8_ds_mla in concat_and_cache_mla, because the model is NoPE MLA. Nothing has merged yet. Of the open PRs the useful one for a Spark is #53969: it has a confirmation from 2026-09-18 on 2x GB10 sm_121 that the model serves at TP=2 and 262K context with that patch applied.
@jahnclawdmonet I appreciate the info! We have had tonyd2wild’s recipe running since release. This setup is being used for a work project, and required to use US based models. Somehow the Nvidia releases of models has been approved so we are diligently trying to find a working recipe to replace our current “non-compliant” version.
The two checkpoints quantize different module sets.
RedHatAI’s NVFP4 group targets one regex, layers.(3-44).mlp.experts.*.(gate|up|down)_proj, and a second group puts layer 45’s experts in block FP8. Its ignore list names the three dense MLPs explicitly, layers.{0,1,2}.mlp.{gate,up,down}_proj. The NVIDIA build targets Linear and ignores self_attn* on every layer, mlp.gate, shared_experts*, model.visual*, embed_tokens and lm_head. Neither the dense MLPs nor layer 45 appear in that list. In the NVIDIA build both are NVFP4, against full precision and block FP8 respectively in what you run now.
The activation scales differ as well. NVIDIA’s input_activations is dynamic: false, RedHatAI’s is dynamic: "local".
Both use Glm5NextForConditionalGeneration, index_topk 2048, index_kpool 4 and qk_rope_head_dim 0. quant_method goes from compressed-tensors to modelopt, which is a different loader. The NVIDIA config also declares kv_cache_scheme FP8 where RedHatAI declares none, and it ships 33 shards against 11.
Do the third party recipes make it that difficult to run 4 sparks or change the underlying model?
I personally use the eugr docker setup with stock vLLM, but with the Intel w4a16 autoround which I find to be the best.
Just either have to patch vLLM so that the flashinfer sparse sm120 backend overcomes the “pe_dim must be 64 for fp8_ds_mla” error, or you can just make a trivial patch to have vLLM select the sm90 backend which doesn’t have the problem.
There is the existing mods setup of the eugr docker which makes it very easy to patch vLLM when the Docker is loading.
With this Docker you can change the configuration easily, including the model or the tensor parallel
Check out my thread here: DGPP: a GB10-optimized C++/CUDA inference engine - #11 by stephen.douglas.hawkins
It’s a different inference engine, but I’m serving GLM-5.3-Flash on a 4-node spark cluster at ~50-60 tokens/sec.
What’s the performance of the model with speculative decoding disabled on TP=2 and TP=4? Single stream and concurrency?
I don’t have concurrency benchmarks with MTP disabled, but for single stream:
TP=2: ~22-24 t/s
TP=4: ~38-40 t/s
I have more detailed benchmark results posted in the repository benchmarks doc
With the way the engine currently performs, MTP1 is the optimal setting. MTP2 provides very little benefits over MTP1 and leads to wider variations (performance swings up and down more). This is because the kernels are basically tuned to keep the spark’s memory controller fully saturated, and each predicted token must be validated against the experts that would have been loaded for that predicted token, meaning the entire “pass” takes longer for higher levels of MTP. so when a prediction is rejected, you pay the cost of that longer pass without the benefit of the accepted tokens.