Xiaomi released MiMo-V2.5-DFlash last week — not a new model, but their 310B MoE backbone bundled with a 2.9 GB block-diffusion draft model (DFlash: fills a block of 8 masked tokens in one forward pass instead of drafting serially like EAGLE). I got it running on my GB10 pair this weekend. Numbers, gotchas, and full reproduction below.
TL;DR: you only need the dflash/ folder from the repo (2.9 GB) — pair it with the NVFP4 community quant you’re probably already running. On 2× Spark, TP=2 eager: 22.3 tok/s with no speculation → 27.6 prose / 45.4 code / 55.1 math / 66.9 JSON with DFlash. Acceptance scales with output structure, which also explains the “low acceptance anomaly” reported elsewhere.
Why the drafter is worth it on Spark specifically
Cross-node TP=2 on the QSFP link means every decode token pays ~48 cross-node all-reduces in eager mode. That latency is fixed per engine step, not per token — so a drafter that gets 3–6 tokens accepted per step amortizes exactly the thing that makes dual-Spark decode slow. This is the same bet as EAGLE3 setups people run here, but DFlash drafts the whole block in one forward instead of token-by-token.
What’s in the repo
- The FP8 backbone (~328 GB, Xiaomi’s pp/ep-sharded format). Ignore it — it doesn’t fit on a pair (2× ~120 GB usable), and requantizing it buys you nothing over the existing community quant.
dflash/— the drafter: 5-layer qwen3-arch, hidden 4096, SWA-1024, block size 8, cross-attends to backbone hidden states from layers [0, 11, 23, 35, 47], plus amask_embedding.ptsidecar (the<|MASK|>vocab row is near-zero; without the sidecar your drafts are garbage — vLLM loads it automatically).
Target backbone: lukealonso/MiMo-V2.5-NVFP4 (~170 GB, fits TP=2 with room for 131K bf16 KV).
Software requirements
vLLM ≥ 0.23.1 nightly with three PRs: #45200 (merged a while back), #45181 (mixed KV page sizes — in nightlies since ~June 21), and #46104 (SWA + DFlash for MiMo — merged July 1, so anything older needs it applied by hand; it applies clean on late-June builds).
On top of the PRs, four things a stock build won’t do (details in the pitfalls section): register the mimo_v2 config type, fix the fused-qkv loader for this NVFP4 export, add SupportsEagle3 to the omni class, and port the PR’s non-causal window fix to the Triton backend.
The serve command
vllm serve lukealonso/MiMo-V2.5-NVFP4 \
--trust-remote-code --dtype auto --kv-cache-dtype bfloat16 \
--hf-overrides '{"architectures":["MiMoV2OmniForCausalLM"]}' \
--speculative-config '{"method":"dflash",
"model":"/path/to/dflash-mimo-v2.5/dflash",
"num_speculative_tokens":7,
"attention_backend":"TRITON_ATTN"}' \
--enforce-eager \
-tp 2 --gpu-memory-utilization 0.83 \
--max-model-len 131072 --max-num-seqs 4 --max-num-batched-tokens 4096 \
--enable-prefix-caching --enable-chunked-prefill \
--enable-auto-tool-choice --tool-call-parser mimo --reasoning-parser mimo
num_speculative_tokens: 7 = the drafter’s block size (8) minus the bonus token. Download just the drafter with:
hf download XiaomiMiMo/MiMo-V2.5-DFlash --include "dflash/*" --local-dir <dir>
(both nodes need it). Env: the usual GB10 NVFP4 set (VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass, VLLM_USE_FLASHINFER_MOE_FP4=1), NCCL_NET_GDR_LEVEL=LOC + NCCL_PROTO=LL for eager decode latency, NCCL_MAX_NCHANNELS=2.
Results (2× GB10, TP=2 cross-node, eager, single stream)
| Config | accept/draft | tok/s |
|---|---|---|
| No speculation (baseline, same stack) | — | 22.3 |
| DFlash — prose, temp 0 | 2.4 | 27.6 |
| DFlash — code, temp 0 | 3.6 | 45.4 |
| DFlash — math (step-by-step), temp 0 | 4.5 | 55.1 |
| DFlash — JSON generation, temp 0 | 6.1 | 66.9 |
| DFlash — prose, temp 0.7 | 1.45 | 24.4 |
| DFlash — code, temp 0.7 | 3.4 | 44.1 |
Vision (omni class) and tool calling both work through the spec-decode path. 2K-token generations stay coherent.
On the “acceptance anomaly”
There’s an SGLang issue reporting accept_length 1.42 vs Xiaomi’s advertised 6.30 for the Pro variant, concluding the released drafter might not match training config. I initially reproduced the same scare — 1.85 on my first benchmark — but per-workload probing shows the drafter is fine: acceptance is just extremely workload-dependent. Free-form prose at nonzero temperature is the worst case (~1.5); structured output hits 6+. Xiaomi’s headline numbers are evidently from structured/reasoning workloads. If your traffic is agents doing tool calls (JSON-heavy), you’ll live in the 3.5–6 band, which is where this thing shines. Benchmark on your workload mix before concluding anything is broken.
Also honesty about attribution: don’t compare against your old stack and credit the drafter for everything. My previous deployment (older vLLM, 1M-context NVFP4-KV pool, MTP1) did 10.6 tok/s; moving to a current build with a lean 131K bf16-KV config got to 22.3 before any speculation. DFlash is the multiplier on top: +24% prose, +2–3× on structured output.
Pitfalls (each one cost me a failed launch)
-
model_type: mimo_v2unknown. Transformers 5.12 doesn’t know MiMo and the NVFP4 export has no remote code. Register a minimalPretrainedConfigsubclass in vLLM’s_CONFIG_REGISTRY(a bare class carrying the checkpoint attrs is enough — vLLM’s model code does the rest). -
Upstream fp8 fused-qkv loader crashes on this checkpoint (
size of tensor a (4096) must match ... (16384)in_shard_fp8_qkv_proj). The NVFP4 export storesqkv_proj.weight_scale_invas MXFP8 E8M0 — U8, shape[rows, cols/32], row-aligned — not the fp8 128×128 block scales upstream assumes, and the fused QKV is laid out canonically[Q|K|V], not Pro-interleaved. Fix: split into q/k/v and delegate toQKVParallelLinear’s own weight_loader instead of hand-sharding. -
Garbage output that loads cleanly.
MiMoV2OmniForCausalLMis missingpacked_modules_mapping, so the ModelOpt mixed-precision config can’t unfusegate_up_projand the dense MLP layers (0–3) silently load MXFP8 packed weights as bf16. Output degenerates after a while and it looks like a sampling problem. Add the mapping to the omni class (andbias=Trueon the vision merger MLP while you’re in there). -
SupportsEagle3on the wrong class. PR #46104 adds the aux-hidden-state interface to the text class. If you serve the omni class via--hf-overrides(you want vision), addSupportsEagle3there too — the protocol’s default methods delegate throughlanguage_model.modeland just work. -
FLASH_ATTN is not valid ... ['attention sinks not supported']. Both the MiMo backbone and the DFlash drafter use attention sinks, and the FA build for SM121 doesn’t support them — everything runs on TRITON_ATTN. But PR #46104’s non-causal sliding-window fix (bidirectional queries need a symmetric window(w,w), not causal-shaped(w,0)) only patchedflash_attn.py. Port the same symmetrization totriton_attn.py’sunified_attentioncall, or your drafter’s block attention is silently wrong. -
Memory headroom is tighter than you think. 170 GB weights + 3 GB drafter + profiling peak at 0.80 GMU left me 0.19 GiB for KV. 0.83 works on a box that also runs other services; do the arithmetic for yours before burning a 10-minute load cycle.
-
First-launch timeout. If you wrap this in systemd,
TimeoutStartSec=3600. Cold loads of 170 GB take a while and the default will kill it mid-load.
Verdict
For a dual-Spark pair serving agent traffic, this is the best MiMo-V2.5 deployment I’ve found: 2–3× decode on exactly the workloads agents generate, vision and tools intact, and the drafter costs 3 GB. The published drafter is healthier than the early reports suggested — measure acceptance on structured output before writing it off.
Happy to share the full recipe/mod scripts if there’s interest.