Minimax3 on 2 nodes decode ~10.7 tok/s, 4bits

[karol.spark]

Sharing a working recipe for running MiniMax-M3 across two DGX Spark (GB10) nodes with llama.cpp layer-split over RPC — including a native tool-calling fix (a hybrid chat template) that isn’t upstream yet.

The full BF16/MXFP8 checkpoint is far too big for a single 128 GB Spark (MiniMax-M3-MXFP8 is ~444 GB), so this splits a sub-4-bit GGUF across two boxes. End result: a stable OpenAI-compatible endpoint with reasoning and function-calling.

The hard part, in one line: llama.cpp PR #24523 adds preliminary M3 support, but its tool-call parser can’t read M3’s native format (HTTP 500). M2’s template parses but corrupts M3’s generation. The fix is a hybrid template — M3 native body + M2’s tool-call format.

Full scripts, the template and the generator are in the repo: GitHub - karolpalys/minimax3-on-2-nodes: Minimax3 on 2 nodes: run MiniMax-M3 426B across two DGX Spark nodes via llama.cpp RPC, with a working native tool-calling hybrid chat template · GitHub

What you get

Metric	Value (UD-IQ4_XS, 2× DGX Spark)
Model	MiniMax-M3 426B MoE, UD-IQ4_XS GGUF (~194 GiB, ~97 GiB/node)
Decode (tg)	~10.7 tok/s
Prefill (pp)	~590 tok/s @ --ubatch-size 2048 (8k prompt)
Context	65,536 (configurable; KV q8_0 ≈ 45 KB/token)
Tool-calling	✅ native structured tool_calls via the hybrid template
Reasoning	✅ <mm:think> separated into reasoning_content
First load	~13–25 min (RPC streams the worker’s layers; cached after)

Setup notes that actually matter

• Two GB10 nodes (or any two CUDA boxes) whose combined memory clears ~194 GiB, i.e. ~97 GiB free per node.
• CUDA + cmake + recent GCC (Sparks ship aarch64 / GCC 13), and ~250 GB free disk for the GGUF.

--split-mode layer puts a contiguous layer range on each device, so there’s exactly one cross-node transfer per micro-batch (at the split boundary). That’s why a bigger --ubatch-size helps prefill: fewer, larger transfers.

Build (both nodes)

git clone https://github.com/karolpalys/minimax3-on-2-nodes
cd minimax3-on-2-nodes
CUDA_ARCH=121 bash scripts/build-llama-cpp.sh      # 121 = GB10 / sm_121

The helper fetches PR #24523 (preliminary M3 support) and merges origin/master to pull in two chat fixes that are required so tool-calling doesn’t hard-crash: #24329 (peg-native parsing) and #24653 (grammar left-recursion). The M3 commit only touches src/models/… and the chat fixes only touch common/chat*, so the merge is conflict-free. Builds with -DGGML_CUDA=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES=121.

⚠️ Worker RPATH gotcha. rpc-server has an empty RPATH, so running it from its own dir fails with libggml.so.0: cannot open shared object file even though the .so are right there. It must be launched with LD_LIBRARY_PATH=~/llama.cpp-bin (the run script does this).

Download the model (HEAD)

bash scripts/download-model.sh                 # UD-IQ4_XS by default

Custom downloader because the large (~49 GB) shards are Xet-backed and hf download can stall at ~0 B/min, while HF_HUB_DISABLE_XET=1 rejects shards > ~48 GB. The script resolves each file to its cas-bridge presigned URL (plain HTTP, byte-range) and pulls with 16 parallel range requests, per-part resume and curl -f. UD-IQ4_XS is the sweet spot that fits two Sparks with headroom; other quants via QUANT=UD-Q4_K_XL.

The hybrid chat template (the key fix)

This is the part that makes native tool-calling actually work.

Problem. M3’s native template emits tool calls with a namespace token before every XML tag plus a recursive argument encoder. llama.cpp’s auto-derived peg-native parser can’t read it:

HTTP 500  "Failed to parse input at pos N: …<tool_call>…"
HTTP 500  "The model produced output that does not match the expected peg-native format"

Trap. Swapping in the MiniMax-M2 template makes the parser happy (M2 uses a simple <minimax:tool_call><invoke><parameter> format), but M2 forces <think> in the generation prompt while M3 was trained on <mm:think>. M3 then degrades — truncated answers (“2+2” → “Simple math question.” then stop) and prose narration instead of tool calls.

Fix — hybrid. Keep M3’s native template unchanged (roles, <mm:think>, adaptive generation prompt, <response> tool-result format) and change only the tool-call format to M2’s parser-friendly style. M3 obeys the in-context format and the parser returns structured tool_calls, while its behaviour stays native. Verified across single call, multi-turn (call→result→answer), and reasoning (no <mm:think> leak into content).

templates/MiniMax-M3-hybrid.jinja is ready to use. To regenerate it for an updated GGUF:

python3 tools/make-hybrid-template.py \
  --gguf .../MiniMax-M3-UD-IQ4_XS-00001-of-00006.gguf \
  --out  ~/MiniMax-M3-hybrid.jinja

It extracts the native template from the GGUF and applies three precise transplants (token defs, the tool-instruction block, the tool_calls renderer), asserting each block matched exactly once so it fails loudly if a future template changes shape.

Run (HEAD node)

WORKER_HOST=10.20.20.2 \
WORKER_USER=$USER \
MODEL=~/models/MiniMax-M3-GGUF/UD-IQ4_XS/MiniMax-M3-UD-IQ4_XS-00001-of-00006.gguf \
TEMPLATE=~/MiniMax-M3-hybrid.jinja \
bash scripts/run-minimax-m3.sh

It kills stale instances, optionally drops page cache on both nodes, starts rpc-server on the worker with the right LD_LIBRARY_PATH, waits for its port, then execs llama-server with the tuned flags. First load streams the worker’s ~97 GiB over RPC (~13–25 min); later loads are faster thanks to rpc-server -c tensor caching.

Key flags:

Flag	Why
--rpc <worker>:50052 --split-mode layer	distribute layers across the 2 nodes
-ngl 999	all layers on GPU
-fa on	flash attention
--ubatch-size 2048 --batch-size 2048	+26% prefill vs default 512 (fewer cross-node transfers); negligible extra memory
--parallel 1	one slot, full ctx to a single request (raise for concurrency)
--cache-type-k/v q8_0	KV in q8_0 to fit memory
--jinja --chat-template-file …hybrid.jinja	the fix
--temp 1.0 --top-p 0.95 --top-k 40	MiniMax-recommended sampling

Pre-flight: no other big model on either node, stop earlyoom (it will SIGTERM llama-server mid-load), and confirm the fast link (ping -c2 10.20.20.2 should be < 1 ms).

Performance & tuning

• Prefill scales with --ubatch-size: 512 → ~469 tok/s, 1024 → ~551 (+18%), 2048 → ~592 (+26%). Diminishing returns past 2048, so that’s the chosen sweet spot. Memory cost was negligible.
• Decode (~10.7 tok/s) is bounded by dual-node RPC serialization and the dense-attention fallback — M3’s native sparse attention isn’t implemented in llama.cpp. --ubatch-size doesn’t affect decode.
• Prompt cache is on by default — repeated prefixes (agent loops) are nearly free on later turns.
• Context: KV q8_0 ≈ 45 KB/token → 64k ≈ 2.8 GiB, 128k ≈ 5.6 GiB per node. Raising ctx eats into the thin per-node headroom.

Troubleshooting

Symptom	Cause / fix
libggml.so.0: cannot open shared object file on worker	rpc-server empty RPATH → launch with LD_LIBRARY_PATH=~/llama.cpp-bin
Tool calls return HTTP 500 (peg-native / Failed to parse)	not using the hybrid template, or llama.cpp lacks the master chat fixes — rebuild and pass --chat-template-file …hybrid.jinja
Truncated / narrated answers, no tool calls	you used M2’s template instead of the hybrid — regenerate with make-hybrid-template.py
llama-server killed mid-load	an OOM killer (earlyoom) — stop it before launch
rpc-server did not come up	wrong worker IP (WiFi vs fast link), SSH not passwordless, or libs missing on worker
hf download stalls at ~0 B/min on big shards	Xet stall — use scripts/download-model.sh (parallel cas-bridge curl)
Throughput drops / RDMA collapse after a reboot	clean second reboot of both nodes (Spark mlx5 WC + nvidia-peermem quirk)

Repo & credits

Full scripts, the hybrid template and the generator: GitHub - karolpalys/minimax3-on-2-nodes: Minimax3 on 2 nodes: run MiniMax-M3 426B across two DGX Spark nodes via llama.cpp RPC, with a working native tool-calling hybrid chat template · GitHub (MIT)

Built on llama.cpp (PR #24523 for M3; #16932, #24329, #24653 for the chat/tool-call path) and unsloth’s MiniMax-M3-GGUF dynamic quants. The contribution here is the hybrid-template approach plus the dual-node RPC recipe that make M3’s native tool-calling work end-to-end without degrading the model’s behaviour.

Generated with AI. Good luck and have fun!

[corbett_korbett]

Thank you! How’s its intelligence since its falling about to dense attention? I would assume that degrades capabilities of the model since it’s not supposed to use dense attention.

[tonyd615]

And this is actuall good Karol because on 3 sparks Working recipe: MiniMax-M3 NVFP4 at TP=3 on 3x DGX Spark (no 4th node) + the OOM fixes - #17 by tonyd615 I still get 10tok/s so more sparks do doesn’t speed it up just more cache and concurrency

[karol.spark]

Dense (full) attention is not a downgrade, it’s the reference computation. Sparse attention is an approximation of full attention introduced for efficiency, not for capability. A model’s sparse-attention scheme is designed so its output stays as close as possible to
what full attention would produce, while skipping most of the O(n²) work. So when llama.cpp falls back to dense attention, the model is getting the complete attention it would otherwise approximate — if anything that’s the more faithful, higher-fidelity computation,
not a lossy one.

What you actually pay for the dense fallback is compute and memory, not intelligence:

Speed: dense is O(n²) vs the sparse ~O(n), so prefill is slower (this is exactly why our prefill numbers are what they are).

Prctical context length: dense KV/compute grows fast, so you can’t cheaply reach M3’s full 1M-token context on this hardware that’s the real reason sparse exists (to scale long context efficiently). We cap context at 64k here partly for that. But within a given
context window, you get full-attention quality.

One honest caveat: there’s a theoretical train/inference mismatch (the model was trained with its sparse pattern, we run dense). In practice the weights are standard softmax-attention weights, so dense is the exact superset and empirically the outputs are fully
coherent: reasoning, multi-turn, and native tool-calling all work correctly in my tests.

If you’re worried about capability loss, the thing to scrutinize is the 4-bit quantization (UD-IQ4_XS), not the attention fallback the quant is a far bigger quality lever than dense vs sparse. Running a higher bit quant (if your memory allows) buys back more quality
than anything attention-related would.

TL;DR: dense fallback costs you throughput and max usable context, not smarts.

[karol.spark]

Yeah, that actually lines up nicely with what I’m getting, good cross check.

And it makes sense once you stop expecting more nodes to help here. Single-stream decode is bound by the per-token sync between nodes plus the model itself, not by how much compute you throw at it, so a 3rd Spark won’t move tok/s, you just get more KV headroom and
room for concurrency/batching. Same conclusion I came to on my side: extra hardware buys you cache and parallel requests, not a faster single stream.

Heads up though, I’m going to be away from the machine for a few days starting tomorrow, so I haven’t finished the full eval yet. Didn’t want to just sit on what I have, so I’m sharing the recipe and findings as is and I’ll pick up the rest of the testing once I’m back

[tonyd615]

If I were to take you recipe and run with it I would get more context but would love concurrency right ? With llama.cp ?

[MiaAI_Lab]

Any way adding MTP on this?

[karol.spark]

Right, but only because I set --parallel 1 in my recipe (one slot, full context to a single request). It’s a knob, not a llama.cpp limit. Bump --parallel N and you get N concurrent slots with continuous batching, the catch is the KV cache gets split so each slot has
ctx/N. So it’s really a trade max context for one request vs more concurrent requests with smaller context each. Pick whichever your workload needs.

[karol.spark]

Not really in a way that pays off here. llama.cpp does speculative decode via a draft model (EAGLE3 included), not the model’s native MTP heads, and I already tried the EAGLE3 drafter for M3, acceptance was only ~37-39% and I got basically zero net speedup. Reason is
the bottleneck on a 2-node setup is the per-token verification sync over RPC, so even accepted draft tokens still pay the cross-node round trip. Spec decode like that helps a lot more on a single node than split across two. If someone gets real MTP heads wired up in
llama.cpp I’d love to retest, but as of now it’s not worth the trouble on this topology

[karol.spark]

Ran the full 715-scenario agentic/reasoning suite twice on Toolery: full-precision via API, and the local UD-IQ4_XS quant on 2× Nodes. Same scenarios, same graders, both runs. Here’s the honest picture, plus a clean full-precision vs 4-bit breakdown at the end.

Full precision (API) vs 4-bit (UD-IQ4_XS, dual-Spark)

Identical 715 scenarios, graders fixed up, tight token budgets on both sides.

Tier	API (full)	4-bit	Δ
easy	93.5%	85.0%	−8.5 pp
medium	90.2%	89.3%	−0.9 pp
hard	70.6%	62.4%	−8.2 pp
very_hard	44.2%	46.7%	+2.5 pp*
TOTAL	78.7%	74.5%	−4.2 pp

* noise, see caveats below

Where it’s strong

• General reasoning & knowledge: medium tier ~90%, easy ~93%. Solid, coherent step-by-step thinking.
• Hallucination resistance: refused to fabricate a citation for a fake paper (no invented DOI or journal). One of its best traits.
• Safety instincts: warns before destructive ops, and in one case actually detected prompt injection in tool results and flagged it instead of blindly following.
• Clean reasoning + tool-calling: thinking separates properly, and multi-turn tool loops complete coherently once the tool format is right.
• Quantizes gracefully: 4-bit keeps ~95% of full-precision quality (only ~4 pp drop overall).

Where it’s weak

• Agentic follow-through: it explores but often doesn’t finish multi-step tasks (greps and reads but never edits+commits; “write a test file” never gets written).
• Degenerate loops: on hard debugging it sometimes repeats one tool, e.g. ran the test suite 8× without ever reading or editing the code.
• Error recovery: doesn’t auto-retry after a tool error or timeout; asks the user or gives up.
• Tool discipline: over-eager calls, weak parallel fan-out (2 of 3 cities), and it sometimes answers from memory instead of using tools to investigate.
• Structured output: leaks reasoning preamble before “JSON only” responses, breaking strict-format checks.
• Hardest-tier collapse: very_hard drops to ~44% (true for both precisions).

Capability ratings

Dimension	Rating	Notes
Reasoning / knowledge	★★★★☆	easy 93% / medium 90%
Hallucination resistance	★★★★★	refuses to fabricate sources
Safety / restraint	★★★★☆	cautions, spots injection; a few edge misses
Tool-calling (basic)	★★★★☆	clean single + multi-turn
Agentic follow-through	★★☆☆☆	explores, doesn’t finish multi-step
Error recovery / retry	★★☆☆☆	no auto-retry on tool failure
Structured output (JSON-only)	★★★☆☆	reasoning preamble leaks
Hardest agentic/coding (very_hard)	★★☆☆☆	~44%
4-bit quant robustness	★★★★★	~95% quality retained

Bottom line

• Absolute drop: 4.2 pp (78.7% → 74.5%).
• 4-bit retains ~94.7% of full-precision quality.

Strong result for a 4-bit quant: Unsloth’s dynamic UD-IQ4_XS loses only about 5% quality versus full precision.

Caveats (honestly)

• The very_hard +2.5 pp (4-bit “beating” the API) is noise: hardest tier, both runs at the floor (~45%), small N, high variance. Don’t read it as a signal.
• The loss concentrates in easy and hard (~8 pp each); medium is almost untouched. So quantization bites hardest on precision in simple tool tasks and mid-difficulty debugging/coding, not on the hardest reasoning, where both precisions struggle anyway.
• 17 “genuine” failures (vh-01/04, restraint, interactive-sudo…) fail on both precisions. Those are M3’s own limitations, not quantization, and they wash out in the Δ.

[karol.spark]

Follow-up: how the dual-Spark M3 (UD-IQ4_XS) setup behaves as you push context length from 64k to the full 262144.

Context-length scaling: 64k / 128k / 256k

Metric	64k	128k	256k (262144)
Status	✅ works	✅ works	✅ loads and serves
KV	q8_0	q8_0	q4_0 + cache-ram 2 GiB (q8 = OOM risk)
Memory, spark1	~107 / 121	109 / 121	111 / 121 (~10 GiB headroom)
Prefill (long prompt)	~590 t/s @8k	359 t/s @113k	227 t/s @239k
Ingest time	—	113k → 5.5 min	239k → 18 min
Needle (retrieval)	trivial	@113k FOUND ✅	@239k inconclusive ❓ (preamble / token-cap)
Verdict	fastest	sweet spot ✅	fits, but slow + retrieval unconfirmed

Takeaways

• 128k is optimal. Verified end-to-end (needle clean), comfortable memory, tolerable prefill. Set as the default.
• 256k is possible but marginal. Only with q4 KV and a trimmed cache; dense O(n²) cuts prefill to 227 t/s (18 min for 239k), and the model handles retrieval worse at this length. For special jobs, not daily use.
• The longer the context, the clearer the cost of the dense fallback: 590 → 359 → 227 t/s. Attention quality is unchanged; you pay in time.

[tocs704]

Sounds like this would benefit from this kernel. flashrt/MiniMaxAI-msa-blackwell - Kernel

[renek]

got this one working with 12tk/s on 2x Sparks on vLLM, i think there is big potential with 2x sparks, anybody already optimizing ?

aquaman164/MiniMax-M3-AutoRound-3.2bit-longctx · Hugging Face

[corbett_korbett]

How’s intelligence and do you know if vision works yet?

[renek]

vision is skipped but you can add it… intelligence good but benches take ages…

[karol.spark]

I have to say, I’m genuinely let down by MiniMax M3. It trips up on basic tool calls, ironically, it performs better on difficult tasks but falls apart on the simple ones. After the first round of testing, I even paid for the official API so I could test it directly from the source, guaranteeing I had the best possible precision. Interestingly, in the overall comparison, it couldn’t even outperform MiniMax 2.7 AWQ or NVFP4.

[corbett_korbett]

Damn thats dissapointing.

[renek]

in my tooling and quality checks m2.7 lacks behind Mimo V2.5… always interesting to see different outcomes but M3 was very good in my tests

[karol.spark]

This comparison evaluates the local deployment on the dual-node setup against the official cloud API, with MiniMax M2.7 AWQ acting as the baseline. The results reveal a surprising performance dynamic, where the older, quantized model outperforms the newer generation in overall scoring.

Model	Deployment	Mean Score	Duration	Speed (Gen t/s)	Top Failure Cause
MiniMax-M2.7-AWQ-4bit	Local (Dual)	81.0%	~1.0h (3588s)	20.0	budget_violated (82)
MiniMax-M3-API	Cloud (Single)	78.7%	~13m (811s)	43.7	budget_violated (95)
MiniMax-M3 (Raw)	Local (Dual)	74.5%	~8.0h (28,710s)	10.2	wrong_tool (84)

1. The “Difficulty Paradox” (Overall Score vs. Task Complexity)

The most striking difference between the generations is how they handle task complexity. While M2.7 AWQ wins the overall mean score (81.0%), its performance degrades drastically on the hardest tasks.

Easy/Medium Tasks: M2.7 AWQ is the undisputed leader, achieving near-perfect scores (97.0% and 93.8%). The M3 models noticeably struggle here, with the local M3 dropping to 85.0% on easy tasks.

Very Hard Tasks: The dynamic completely flips. The local MiniMax-M3 excels here, scoring 46.7%, followed by the M3 API at 44.2%. M2.7 AWQ falls significantly behind at just 36.7%. M3 is clearly over-indexed for complex reasoning at the cost of basic reliability.

2. Tool Calling Reliability (wrong_tool Errors)

There is a massive regression in basic tool-calling accuracy with the M3 architecture when run locally.

Local M3: Failed 84 times due to wrong_tool, making it the primary cause of failure.

M3 API: Reduces this issue by half (47 wrong_tool errors), suggesting the cloud deployment benefits from better precision or under-the-hood system prompts.

M2.7 AWQ: Highly reliable with tools, registering only 30 wrong_tool errors. Its primary failure mode is simply running out of steps/tokens (budget_violated).

3. Execution Speed and Hardware Efficiency

Running M3 locally on the dual cluster introduces severe performance bottlenecks compared to the M2.7 AWQ quantization.

Local M3 dragged on for nearly 8 hours (28,710 seconds) at a sluggish 10.2 t/s. It also registered 10 timeout errors.

M2.7 AWQ completed the identical 715-trial scenario in under an hour at 20.0 t/s (C = 4), doubling the throughput of the local M3 while maintaining better overall accuracy.

4. Local Raw vs. Cloud API (The M3 Gap)

Running M3 via the official API resolves several issues seen in the local raw deployment. The API yields a higher score (78.7% vs. 74.5%), cuts the wrong_tool errors almost in half, and eliminates timeout errors entirely. However, even with the API’s advantages, it still fails to beat the overall accuracy of the locally hosted M2.7 4-bit quant.

M2.7:

M3 local:

M3 API:

[nerhun]

For all intends and purposes, this is an unusable quantization for me.

I tried the same quantization (minus tool fixes) of MiniMax-M3 as an assistant and stressed it with debugging tasks where most other models I tried fail (including Opus 4.7 and 4.8). On many tasks it hallucinated command line switches that do not exist, refused to be steered, even refused to accept alternative solutions that I told him to be working.

So far the strongest 2-node candidate for me is the unquantized deepseek-ai/DeepSeek-V4-Flash which seems to be punching above its weight at real 30+ t/s.