MiniMax-M3-AWQ on 4× GB10, fp8 KV, 262k context, adaptive reasoning, ~30 tok/s

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Running cyankiwi/MiniMax-M3-AWQ-INT4 on 4× GB10 with vLLM: fp8 KV cache, corrected long/dense-context output, EAGLE3 speculative
decoding, adaptive reasoning. Patches + recipe below.

Throughput — 4× GB10, TP=4, EAGLE3 k=2, llama-benchy (pp=2048 / tg=512, median of 5).
fp8 KV ≈ bf16:

depth prefill (PP) decode (TG)
0 1579 tok/s 32.2 tok/s
8k 1698 tok/s 29.9 tok/s
32k 1665 tok/s 28.6 tok/s
64k 1596 tok/s 25.5 tok/s

fp8 KV capacity win:

KV dtype KV cache @ 262k max concurrency @ 262k
bf16 ~0.69M tok 2.6×
fp8 1.38M tok 5.27×

Configuration: 4× GB10 (sm_121 aarch64), vLLM 0.22.1 + CUDA 13, TP=4, fp8 KV, EAGLE3
k=2, 262k ctx. Requires vLLM with PR #45744 (fp8 sparse-GQA) + 4 small patches (indexer
fix, reasoning PR #45718, tool parser, NCCL 2.30.4).

Recipe and setup: GitHub - CosmicRaisins/minimax-m3-awq-gb10: MiniMax-M3-AWQ on GB10/DGX Spark (sm_121): vLLM patches + recipe — fp8 KV (~2x context), long-context indexer fix, EAGLE3 spec-decode, adaptive reasoning · GitHub

I used it to run a few front-end tasks today, and Pi reported ~25 tok/s in general. It still felt kind of lazy, in the sense that it usually fails to gather all relevant context and just makes assumptions. So far, I don’t think this is worth it over glm 5.2. Unless you absolutely need the 1M ctx.

2

Nice writeup,
▎
8x GB10 cluster here, couple things to share/ask that are relevant to the numbers.

Networking: nodes are on 200G (ConnectX-7) through a pair of MikroTik CRS804-4DDQ,
That “more nodes doesn’t help single stream” rule was all measured on slower fabric afaict, so at 200G the per-layer sync should be cheaper and maybe it scales. Will post numbers when I have them, haven’t seen anyone bench 8 nodes at this fabric.

▎ On EAGLE3 — you’re at k=2 linear. Did you ever push tree drafting / multiple drafters? Decode’s running maybe a third of the bandwidth ceiling and compute is basically idle, so it looks like you could verify a much wider candidate tree for nearly free and the only real limit is acceptance. Been thinking about a SpecInfer-style multi-drafter tree, sized to the GB10 compute/bw ratio (Sequoia-ish). Curious if you went past k=2 and acceptance just fell off, or if it was out of scope.

Will report back the 8-node / 200G numbers once it’s up. Thanks for the patches.

3

I’m using CRS504 so I’m only at 100G. I’d love to see if 200G makes a difference.

I did try k=3 and acceptance fell off hard and slowed down TG. I did wanted to try tree drafting but I wasn’t sure about vllm support so it was out of scope. I’ll take a look at them today!

4

Why did you need a pair of crs804-4ddq for 8 nodes? With 4x400gb ports using breakouts you can have 8 nodes using just one switch each at 200Gb/sec. Am I missing something here? Also some numbers would be very nice to see so we can compare. Thanks.

5

sorry I’ve been away. right now on tp=8 we are seeing about 47 tps on average. The pair of crs804-4ddq is for other connections to the fabric. you can have 8 spark nodes on one switch but that is every high speed port.
EAGLE3 TP=8 k=2 (small prompt)

Single-stream decode 47tok/s

Prefill 1930 tok/s

8 concurrent 105 tok/s aggregate

NO EAGLE3 TP=8 k=2

Single-stream decode 21 tok/s

working on some other speculative heads now.

-For anyone wondering i really like M3 as an LLM, we have a fairly complex setup with multiple LLM’s running inside a custom memory and communication build, but M3 does a great job. I preferer it over Deepseek V4 and GLM 5.2. That said we are also using Opus and 5.5 but for what M3 is doing it’s fantastic, much better than just judging by the stats

Setting Single TPS 8x aggregate Mean stream TPS P95/tail stream time
k=2 46.8 105.8 18.8 30.2s
k=3 46.9 106 19.1 30.2
k=4 55.2 101.8 17.8 31.4

actually after much testing on many different sizes of context these above numbers are optimistic and only for smaller prompts. We were seeing about 37 tok/s on real life work with k4 the acceptance rate on large context drops down to nearly nothing. We are sticking with K2 for now and seeing around 38 tok/s for real life work. K4 is fine on small prompts but in real life it doesn’t work out. We are also looking at dflash, anyone else building for p-eagle or dflash?

6

What do you like better about M3 over GLM 5.2? Speed and vision?

7

i don’t use M3 for heavy coding, but for my use i like they way it communicates, shorter non verbose, gets to the point, seems very good at longer projects staying on track, just a nice clean model, it reminds me of Gemini 4 which i really like but the api costs with a large persistent context window is a killer. I’ve been testing M3 on k3 and k4, I’ll probably switch to k4. Also working on P-Eagle heads. Also for it’s total size it’s quite impressive, I only have so much experience but it’s currently my favorite open weight model

8

we are seeing a decent speed increase with eagle tp=1 instead of eagle tp=8

9

How are you enabling draft tp=1?

10

For M3, we enable draft TP=1 through vLLM’s EAGLE3 speculative config:

--speculative-config '{
  "method": "eagle3",
  "model": "Inferact/MiniMax-M3-EAGLE3",
  "num_speculative_tokens": 2,
  "draft_tensor_parallel_size": 1,
  "attention_backend": "TRITON_ATTN"
}'

In our M3 recipe it is launched like this:

vllm serve cyankiwi/MiniMax-M3-AWQ-INT4 \
  --served-model-name minimax-m3-awq \
  --trust-remote-code \
  --block-size 128 \
  --attention-backend TRITON_ATTN \
  --kv-cache-dtype fp8 \
  --language-model-only \
  --distributed-executor-backend ray \
  -tp 8 \
  --speculative-config '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3","num_speculative_tokens":2,"draft_tensor_parallel_size":1,"attention_backend":"TRITON_ATTN"}'

The key field is:

"draft_tensor_parallel_size": 1

That keeps the small EAGLE3 draft model from being sharded across all 8 GB10s.

for GLM 5.2 passing it through vLLM’s speculative config JSON:

--speculative-config '{
  "model": "/root/.cache/huggingface/hub/models--CosmicRaisins--GLM-5.2-MTP-INT4-aligned/snapshots/<rev>",
  "method": "mtp",
  "num_speculative_tokens": 2,
  "draft_tensor_parallel_size": 1,
  "attention_backend": "FLASHMLA_SPARSE"
}'

In our launcher it’s built here:

SPEC_CONFIG=$(printf '{"model":"%s","method":"mtp","num_speculative_tokens":%s,"draft_tensor_parallel_size":%s,"attention_backend":"FLASHMLA_SPARSE"}' "$DRAFT_MODEL" "$SPEC_TOKENS" "$DRAFT_TP")
SPEC_ARGS=(--speculative-config "$SPEC_CONFIG")

Then launched with:

SPEC_TOKENS=2 DRAFT_TP=1 ./glm52-launch-awq-tp8.sh

Caveat: this depends on the vLLM build exposing draft_tensor_parallel_size in SpeculativeConfig. Our build is v0.23.1rc1.dev101+g4c6266331.d20260626.

11

Have you benchmarked how cyankiwi/MiniMax-M3-AWQ-INT4 performs in terms of accuracy vs cloud MiniMax M3 or an FP8/NVFP4 quant?

12

Not yet — we’re moving to the NVFP4 quant later today and will test (accuracy + throughput) against the AWQ-INT4 build, have you?

13

Not yet, just bought the 4th spark and a switch but annoyingly the spark came with the wrong power adapter. Should be good to begin setting things up and testing tomorrow. I’ll give this a go and the Nvidia NVFP4 quant based on the sparkrun recipe.

14

power adapters have been an issue for me also, they fail but not a full failure, it seems it will boot then an hour or so later the power adapter fails. There is an aftermarket one on amazon that I’ve been buying. I wish there was a power brick with 8+ cords coming out of it instead of separate power bricks. In general for accuracy it seems really good, M3 is one of the best models I’ve seen for non hallucination and working with other llm’s well. I’m really trying to get speed around 50 tok/s is what i want, but it’s a hard road. good luck

15

We love M3 so we’ve been working real hard on fine tuning, trying pretty much everything we can think of. right now this is were we stand, let me know if you have any questions. these numbers are run through benchy, we use m3 for customer chat and some tool calling, so we focus on pp=100 and pp=2048 plus a lot of concurrent sessions.

MiniMax-M3 (456B MoE, AWQ INT4) on 8× DGX Spark

TP=8 over 200GbE RoCE · EAGLE3 k=2 (65.5% acceptance, 2.31 tok/step) · vLLM + Ray · fp8 KV
Measured with llama-benchy, quiet server, 3 runs/level. 0 errors at every level.

128-token prompt (realistic chat turn)

Users Agg tok/s Per-user tok/s TTFT
1 51.8 51.8 0.46 s
2 59.6 43.7 0.61 s
4 87.4 36.8 0.58 s
8 126.2 27.5 0.87 s
16 172.5 16.4 1.54 s
32 188.1 10.8 2.60 s
48 179.1 8.8 5.87 s
64 188.1 8.2 9.89 s

2048-token prompt (document / RAG turn)

Users Agg tok/s Per-user tok/s Prefill tok/s TTFT
1 42.3 42.3 1913 1.13 s
2 62.2 34.1 1847 1.66 s
4 98.0 29.7 2060 3.49 s
8 104.0 19.9 2096 6.94 s
16 123.3 10.7 2139 11.9 s
24 111.5 6.7 2131 16.7 s
32 137.5 6.9 2144 20.7 s

Engine saturates at ~190 tok/s, gpu_memory_utilization: 0.70 is mandatory (a bug with how vllm recognizes gb10 memory. we have a few more ideas but this is the best recipe yet for us, we use this is a production environment. note cuda graphs off, we had issues with them and max_num_seqs:32. We are chipping away at latency now, would love to try this next TP=4 + PP=2 halves the ring. That’s the PP-PATCH.md path, still blocked on the SGLang #10272 issue. there is still room for faster tokens, it’s all about latency(Not Network, kernels(~25.8 ms or ~58%)) right now, ~85% of every step is collective latency

16

For people struggling with nvfp4 this is really important, we came to this conclusion and then found this great website: [Benchmark] NVFP4 Is a Trap on GB10: FP8 Wins by 32% (vLLM + SGLang Tested) — ai-muninn the key is this: GB10’s SM121 GPU lacks the cvt.rn.satfinite.e2m1x2.f32 PTX instruction for native FP4↔FP32 conversion. SM120 (RTX 5090) and SM100 (B200) have it in hardware. On SM121, NVFP4 either falls back to Marlin dequant (40.8 tok/s, -21%) or crashes. This is a silicon-level limitation — no driver or firmware update can fix it.

Possibly everyone already knows this, i didn’t and found the hard way, hope this can be helpful to someone else.

17

Note (2026-04-30): NVIDIA later clarified on the developer forum that this instruction actually does exist on GB10 — it requires the sm_121a target rather than sm_121. The observation in the original article was real (under sm_121 it’s missing), but attributing it to “no circuit exists” was wrong.

The moment I read that note on the same post, I stopped reading it myself and sent Fable to read it.
… And the epilogue confirms the conclusion is now stale: the same blog’s later posts show NVFP4 beating FP8 by ~1.5× for single-stream decode on a dense model (as a bandwidth/compression win) and NVFP4 W4A4 hitting 67 vs 52 tok/s against FP8 on an MoE once CUDA graphs were enabled. [Benchmark] NVFP4 W4A4 beats FP8 on a DGX Spark MoE: 67 vs 52 tok/s once CUDA graphs fire — ai-muninn

18

Thanks for that tip, we had come to that conclusion then found the site and gave up. You are absolutely correct though, it does run with nvfp4. We had tested both ways and found that nvfp4 was still not as fast as our fully tuned eager int4 build. We also worked on training dflash heads at 5k, 50k and 250k. c=1 was quite a bit lower than eagle but held up better as C increased, we went back to our eagle3 build