MiMo-V2.5 Omni · TP=2 · 1M context · NVFP4 KV on 2× DGX Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

We unlocked MiMo V2.5 on 2x DGX Spark at FULL 1M Context
Got MiMo V2.5 running on dual DGX Spark with NVFP4 quantization and KV cache optimization. This is the full setup, configs, and deployment pipeline.
What we did:
NVFP4 quantization for efficient inference on Spark hardware
KV cache tuning for multi-turn agent workloads
Dual-node orchestration across 2x DGX Spark instances
Full Hermes Agent integration for autonomous workflows
Why it matters: This proves you can run serious local inference on consumer-accessible DGX hardware without needing datacenter GPUs. The quantization keeps VRAM in check while maintaining output quality.
Repo: GitHub - tonyd2wild/MiMo-V2.5-TP2-1M-NVFP4-KV-2xDGX-Spark: MiMo-V2.5 Omni TP=2 on 2x DGX Spark · 1M context · NVFP4 4-bit KV (~1.01M-token pool, ~30 tok/s) · 69-eval: thinking-OFF 97.8 beats thinking-ON 90.6 for tool/agent work · GitHub
Everything’s documented. Fork it, break it, improve it.

Shoutout to @renek for his recipe work as well and getting this on NVFP4 quality did not degrade

2

You really are too wild… I seriously want to spin this up right now. I have a feeling it’s going to take the rest of my weekend up playing with this lol

3

Thanks!

4

currently working on optmizing the MTP Kernels as well as I want to get more into the details, seems there is still some room to improve…

5

GitHub repo updated

6

Where is you repo

7

Building a version of Tony+Rene mods on top of 0.23dev1 which has 33% of the required mod already merged. Plus no-ray, that’s 10GB ram freed. Wish me luck. Will report here.

8

Trying to build you repo, but you don’t mention which image you use as a base. 0.23 diverges in various parts.

9

my repo for mimo optimized

10

Can you send me your REPO as is even if it is messed up I can have my agent looka t it.

11

But your readme already references his repo

12

who ?

13

No repo, local. But I would like ask which vllm image you used as a base, unless Claude wants to protect this secret too :)
Everything builds and boots but worker blows up on loading tensor, gets to 121 out of 122 gb ram and early oom kills it. Same with eager, minimal cuda graph profiling 1, 64k cache, 0.78 ram. It just balloons ram and dies. I suspect that 0.21 vllm is not what it needs to be

14

GitHub - tonyd2wild/MiMo-V2.5-TP2-1M-NVFP4-KV-2xDGX-Spark: MiMo-V2.5 Omni TP=2 on 2x DGX Spark · 1M context · NVFP4 4-bit KV (~1.97M-token KV pool @ 1M, ~30 tok/s) · 69-eval: thinking-OFF 97.8 beats thinking-ON 90.6 for tool/agent work · GitHub try this bro

15

Yeah I was struggling with it too. After some digging the image must have vllm pr41797. Won’t work otherwise and there is no publically avail spark-vllm image with it. Tony just uploaded his own local image that should work.

I’m compiling the image from scratch anyways.

16

Have you tested it against a q8 Qwen 3.6 27b with f16 kv cache? Not fp8 since the fp8 is typically much dumber than q8. I often find the 27b outperforms every minimax quant I have tried across dual sparks which has me wondering if the 4bit quantized large models are even worth it at this point. Just too much quality loss.

17

I tested on Mac quality of fp8 vs mxfp4 extensively for 27b. Never tested fp8 on spark, even Nvfp4 is too slow, fp8 is unusable for me. Anyways, I also thought that fp8 should be better but now, quality tests indicated parity and mxfp4 beats on speed so fp8 was evicted. Cache wise, rotor quant ir turbo quant with step quantization show no quality loss compared to fp8 or bf16 cache.

Tested on my own task set, data, consistent. Not general tests.

27b is awesome model for specific tasks, it’s great when given all the information needed and does not have to deduce anything. I use it as code auditor/consultant on Mac in my hybrid system, when ds4f starts running in circles trying to solve problem, but only specifically for existing code and full info. It has no enough parameters to effectively construct own, unless something basic or typical it was trained on. And I use qwen 3.5 122b also as a reviewer and second opinion for arch, tech doc, code review as a model with much broader knowledge.

However in general tests the model to beat is Qwen 3.5 122b int4 autoround (I know lol) that consistently marks 90-91 on tool eval hard mode 89 tests. Ds4f takes 86-89 depending of phase of the moon, 27b 84-86, step fun lower. Nemotron 3 super is hopeless at 77, same a speed monster cascade 2.

So why ds4f and not 122b as a main model that occupies 2 sparks in a cluster? Simple : massive world knowledge thanks to 297b parameters, it’s as good as sonnet or gpt 5.4 mini no joke and 1m sessions. I see 400-600k sessions every day, sometimes a couple in parallel. I boot agent with instructions to read chunks of big wiki so the boot immediately loads like 100-120k tokens and then extra info pull, tool calls quickly push past 300k in first hour, then growth slows.

18

That’s what I used. Where I diverged: I tried to use vllm distrubuted, instead of ray, to avoid big ray overhead on main node. But it’s worker that dies and Ray config is easier on worker. I suspect that this is a main divergence. I know that Claude keeps hallucinating in repos saying mp is for a single node (like it has 2 gpus or something) but it’s common. Before I bought a second spark I was asking sonnet and chatgpt about the setup and options and they kept telling me that ray is only option (and it’s not even included in latest vllm releases) and tp=2 will never work as it’s designed to work within the same box multigpu and pp=2 is the only option, and 200g roce will kill any speed and yada yada. With highest confidence. So not surprise.

19

Got this!

Thank you @tonyd615 - this will be my main model now!

Here are the exact fixes, including several that aren’t in the README.

1. OOM at load (“gets to 121/122 GB and earlyoom kills it”) — @0rand

That’s earlyoom killing vLLM, not a vLLM bug. Run systemctl stop earlyoom before launch (or raise its -m threshold). It runs near-ceiling, so the recipe’s stability env vars matter:

  • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  • RAY_memory_monitor_refresh_ms=0 (stops Ray’s 95% monitor from false-killing the head after warmup)
  • Cap the Ray object store to 1 GiB per node: ray start … --object-store-memory=1073741824. Uncapped, Ray reserves a big plasma store and steals unified memory, which OOMs.

2. Which base image — @0rand

Don’t build on 0.23 (it diverges). Use Tony’s published image directly:

ghcr.io/tonyd2wild/mimo-v2.5-tp2-1m-nvfp4kv:20260620

It’s the patched vLLM dev build (0.21.1rc1.dev85+gd87ee1893) and the 6 required mods are vendored in recipe/mods/. Apply them with apply-mods.sh after the container is up. A stock pip install vllm will reject --kv-cache-dtype nvfp4 / triton_attn_diffkv.

If a node can’t pull from ghcr (DNS/egress), pull on the node that can and ship it over your fast link:

docker save … | ssh <other-node> docker load

3. Ray vs. vLLM-distributed worker crash — @0rand

For 2 nodes you must use Ray. The mp (multiprocessing) executor is single-host: it can’t span two boxes, so that worker crash is expected. (Re the “TP=2 only works single-node” point: TP=2 across two boxes works fine, but only via Ray, not the mp executor.) Keep --distributed-executor-backend ray.

4. Gotchas not in the README

These are very likely your “worker still crashes”:

Raylet aborts with Timed out waiting for file …/metrics_export_port_…: the Ray dashboard agent hangs ~20 s on socket.getfqdn() (reverse-DNS timeout) and the raylet’s ~15 s wait fires first. Fix: add the hostname to /etc/hosts inside the worker container: echo "127.0.0.1 $(hostname)" >> /etc/hosts. The agent then registers in under 1 s.

gloo … Unable to find address for: <iface> during init_distributed_environment: gloo picked a DOWN or wrong NIC. Pin gloo and NCCL to your RoCE interface on both nodes: export GLOO_SOCKET_IFNAME=<roce-iface> NCCL_SOCKET_IFNAME=<roce-iface> (on our Sparks both nodes name it enP2p1s0f0np0; check ip -br addr).

Placement group never fills: No available node types can fulfill resource request {'GPU':1.0,'node:10.0.0.x':…}: vLLM auto-detected the wrong host IP (default-route NIC, not RoCE). Export VLLM_HOST_IP=<this node's RoCE IP> before ray start and before vllm serve (head = head RoCE IP, worker = worker RoCE IP).

Start order: worker container → Ray head → Ray worker (join) → wait for 2.0/2.0 GPU → launch vllm serve from the head.

5. NVFP4 vs. Q8 quality — @corbett_korbett

In our tool/agent testing the NVFP4 weights + NVFP4 KV were effectively lossless vs the AWQ-int4 baseline, and the NVFP4 KV is exactly what buys the 1M context (KV pool ~2M tokens at gpu-util 0.84). Solid but task-dependent, so run your own eval.

Results: ~28–30 tok/s decode, 1M context verified

20

Is it written by Ai again? Lol