Mimo V2.5 Flash on 2 Nodes

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I wanted to make a new post on this topic because I want to actually try to optimize this model. With all the models dropping I think Mimo will probably drop a updated version soon but Mimo V2.5 if multimodal and has 1Million Ctx and it can fit on 2 sparks it’s near the edge but it works but it has a few issues. I want to get together and find out what we need to do to get this thing optimized I think I clocked on 25toks but it stopped working. MiMo-V2.5-NVFP4 on 2x Spark Cluster - Recipe, findings, fixes, benchmarks - #31 by renek The original post. If we can get this smoother it will be best than Deepseek v4 flash.

2

They said they’re going to release the DFlash integrated version for the non pro model soon, so you might want to hold off for that.

3

Who is they ? Can you link me 🙏🏽

4

Xiaomi. They’ve already done it for their big model: https://mimo.xiaomi.com/blog/mimo-tilert-1000tps

Near the bottom they say: UltraSpeed support for MiMo-V2.5 is on the way — stay tuned

5
6

Has anyone tested its vision capabilities? I cant find a model yet that can accurately caption videos I have tried every combo of the smaller models with no luck.

7

But when haha

8

I have now 31 tk/s and it’s my daily driver with Hermes

9

recipe ?

10

say what? share details pls

11

prepped by Claude 4.8 Max

Running MiMo-V2.5 (310B NVFP4 Omni) at ~30–33 tok/s on 2× DGX Spark (GB10, SM 12.1, TP=2)

After a lot of iteration, MiMo-V2.5-NVFP4 serves stably on a two-node DGX Spark cluster (2× GB10) over vLLM, TP=2. ~30–33 tok/s single-stream decode, ~57–63 tok/s aggregate at 3 concurrent streams. It does not run out of the box on SM 12.1 — here’s the full recipe.

Hardware / software

  • 2× NVIDIA DGX Spark (GB10), SM 12.1, 128 GB unified each; RoCE/QSFP for the TP all-reduce
  • Model: lukealonso/MiMo-V2.5-NVFP4 — 310B MoE, NVFP4 experts + MXFP8 dense + Omni tower (~86 GB/rank resident)
  • vLLM dev114 line built with PR #41797, CUDA 13.2, TORCH_CUDA_ARCH_LIST=12.1a
  • Driver note: 595.71.05 gives a smaller KV pool (~233K tok) than 595.58.03 (~368K); no throughput difference

The core blocker on SM 12.1

Stock MiMo-V2 forces the FlashAttention-DiffKV backend, which asserts FA3 for attention sinks — GB10 only has FA2 → first inference crashes. Fix: run the Triton DiffKV backend from vLLM PR #41797:

--attention-backend triton_attn_diffkv

The 3 fixes that were all required

  1. Triton DiffKV (PR #41797) instead of FlashAttention DiffKV.
  2. MXFP8 dense dispatch: MiMo is a mixed quant (NVFP4 experts + MXFP8 dense). The modelopt_mixed path didn’t route the dense layers through MXFP8 kernels → garbage logits. Patch the modelopt-mixed handling.
  3. Quantized KV on DiffKV: TRITON_ATTN_DIFFKV raises NotImplementedError on any quantized kv_cache_dtype, but the underlying store kernel already accepts dtype + scales. The guard is defensive — disable it to run --kv-cache-dtype fp8_e4m3, which roughly doubles the KV pool.

Plus, for the Omni multimodal path: serve the absolute snapshot path (not the HF repo id), or the audio_tokenizer loader breaks; and don’t skip the audio_tokenizer. weights.

Working configuration

tensor_parallel_size      2

gpu_memory_utilization 0.89 # hard ceiling — 0.90 fails the GB10 startup guard

max_model_len 163840 # 160K

max_num_batched_tokens 8192

max_num_seqs 6

block_size 32

kv_cache_dtype fp8_e4m3

attention_backend triton_attn_diffkv

speculative-config {"method":"mtp","num_speculative_tokens":2}

--enforce-eager # CUDA graphs blow the KV budget at 160K

--enable-prefix-caching --enable-chunked-prefill

--load-format instanttensor

--enable-auto-tool-choice --tool-call-parser mimo --reasoning-parser mimo

--distributed-executor-backend ray

Relevant env:

TORCH_CUDA_ARCH_LIST=12.1a

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass

VLLM_USE_FLASHINFER_MOE_FP4=1

VLLM_FLASHINFER_MOE_BACKEND=throughput

NCCL_CUMEM_ENABLE=0

NCCL_NVLS_ENABLE=0

NCCL_NTHREADS=8 NCCL_NSOCKS_PERTHREAD=2 NCCL_BUFFSIZE=8388608

VLLM_USE_RAY_V2_EXECUTOR_BACKEND=0 # each "1" costs ~12 GiB on node 2 → freeze

VLLM_USE_RAY_COMPILED_DAG_OVERLAP_COMM=0

Notes:

  • MTP = 2 — model is trained for 2 speculative tokens; MTP=3 is ~6% slower. Acceptance: pos-0 ≈ 86%, pos-1 ≈ 45%, overall ≈ 65%. This is what gets single-stream to ~30 (vs ~14 without).
  • --enforce-eager — CUDA graphs don’t fit the KV budget at 160K.
  • NCCL on GB10: newer NCCL (v2.30u1) reserves a ~7.5 GiB CGA buffer that pushes the startup check over; use a GB10-targeted NCCL build + NCCL_CUMEM_ENABLE=0.

Performance

  • Single-stream decode: ~30–33 tok/s
  • 3 concurrent streams: ~57–63 tok/s aggregate
  • KV pool ~233K tokens (fp8 KV); theoretical roofline on 2× GB10 ≈ 52 tok/s, so single-stream is ~60% of roofline — MTP is the lever that gets it there.

All Omni modalities work (image/video/audio); tool-calling works with --tool-call-parser mimo.

12

Going to tinker with this now

13

how is it going ? any optimizations i missed ? :)

14

going to run this today I passed out also what was you max context you tried ?

15

you can go to max single stream from above info but i need concurrency… with older Nvider firmware you could go up to 300K

16

GitHub - tonyd2wild/MiMo-V2.5-NVFP4-2x-DGX-Sparks-TP-2: Tuned recipe: Xiaomi MiMo V2.5 (NVFP4) on 2x DGX Spark, vLLM TP=2 over RoCE. ~32-33 tok/s, Quality 89.9, 160K context, omnimodal + tool-calling. Non-eager + MTP=2. · GitHub we got it going.a little faster up to 38 tok/s

17

Also tinkered myself a bit and got 39tk/s with synthetic tests but real world only around 33tk/s, I guess thats the cap till we get dflash

18

Might of got a breakthrough might need to make a new post on this one will find out in the next few moments

19

found something ?

20

i have now NVFP4 KV Cache enabled, with 2 Nodes over 500K KV Cache, further optimizing, later i will post the image for easy replication