Thinking about adding a 2nd Spark, so wanted to get a rough idea of performance it would yield for running this.
Same. In for updates.
I’ll try this one, curious how that would compare to AWQ quants. I get 40 t/s from cyankiwi AWQ quant, but the speed degrades VERY quickly with context, much more than any other model I have. Same for QuantTrio AWQ quant, but I only get 28 t/s with 0 context. Didn’t have this problem with M2.
One other thing, though - this is w4a4 quant, so expect it to be “worse” quality wise than w4a16 which most AWQ quants are. Also, not sure how this was quantized and what calibration dataset was used (if any) - I wish people who make quants provided more information about their quantization process and some perplexity metrics, at least. For Minimax M2.1 vLLM-compatible quants, only cyankiwi/MiniMax-M2.1-AWQ-4bit · Hugging Face provides such metrics.
OK, here we go:
vllm serve lukealonso/MiniMax-M2.1-NVFP4 \
--trust-remote-code \
--port 8888 --host 0.0.0.0 \
--gpu-memory-utilization 0.7 \
-tp 2 --distributed-executor-backend ray \
--max-model-len 131072 \
--load-format fastsafetensors \
--enable-auto-tool-choice --tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2_append_think \
--no-enable-prefix-caching
I only use --no-enable-prefix-caching for the benchmarks, especially parameter sweeps. For normal use, don’t include it.
Also it crashes if you specify parameters from the model card, related to flashinfer backend), but the above command works just fine.
vllm bench serve --backend vllm --model lukealonso/MiniMax-M2.1-NVFP4 --endpoint /v1/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1 --port 8888 --host spark
============ Serving Benchmark Result ============
Successful requests: 1
Failed requests: 0
Benchmark duration (s): 3.65
Total input tokens: 12
Total generated tokens: 119
Request throughput (req/s): 0.27
Output token throughput (tok/s): 32.65
Peak output token throughput (tok/s): 34.00
Peak concurrent requests: 1.00
Total token throughput (tok/s): 35.94
---------------Time to First Token----------------
Mean TTFT (ms): 148.96
Median TTFT (ms): 148.96
P99 TTFT (ms): 148.96
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 29.63
Median TPOT (ms): 29.63
P99 TPOT (ms): 29.63
---------------Inter-token Latency----------------
Mean ITL (ms): 29.63
Median ITL (ms): 29.36
P99 ITL (ms): 32.45
==================================================
============ Serving Benchmark Result ============
Successful requests: 10
Failed requests: 0
Benchmark duration (s): 43.49
Total input tokens: 1354
Total generated tokens: 2665
Request throughput (req/s): 0.23
Output token throughput (tok/s): 61.28
Peak output token throughput (tok/s): 96.00
Peak concurrent requests: 10.00
Total token throughput (tok/s): 92.41
---------------Time to First Token----------------
Mean TTFT (ms): 704.69
Median TTFT (ms): 897.09
P99 TTFT (ms): 897.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 88.44
Median TPOT (ms): 89.61
P99 TPOT (ms): 123.78
---------------Inter-token Latency----------------
Mean ITL (ms): 68.09
Median ITL (ms): 59.08
P99 ITL (ms): 125.94
==================================================
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Benchmark duration (s): 134.77
Total input tokens: 22815
Total generated tokens: 21113
Request throughput (req/s): 0.74
Output token throughput (tok/s): 156.67
Peak output token throughput (tok/s): 345.00
Peak concurrent requests: 100.00
Total token throughput (tok/s): 325.96
---------------Time to First Token----------------
Mean TTFT (ms): 5441.25
Median TTFT (ms): 5009.22
P99 TTFT (ms): 9659.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 325.51
Median TPOT (ms): 286.11
P99 TPOT (ms): 808.69
---------------Inter-token Latency----------------
Mean ITL (ms): 239.88
Median ITL (ms): 246.39
P99 ITL (ms): 816.64
==================================================
Done the parameter sweep - still seeing the same bad regression in performance, but I need to make sure I do it right before I post the results.
thanks for running these. Is my understanding correct that NVFP4 is still not fully optimized on GB10?
Yes, that’s correct. For the time being, AWQ quants will give you better performance. Hopefully, proper NVFP4 support comes soon - seems like it requires more than recompiling from the source with the right architecture.
Where does the AWQ plane out in the 65-131K context for a single request? Ballpark is fine, I’m not asking for a sweep.
Here is the full sweep using my new benchmarking tool:
| model | test | t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 | 3544.10 ± 37.29 | 688.41 ± 6.09 | 577.93 ± 6.09 | 688.45 ± 6.10 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 | 36.11 ± 0.06 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d4096 | 3150.63 ± 7.84 | 1410.55 ± 3.24 | 1300.06 ± 3.24 | 1410.58 ± 3.24 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d4096 | 34.36 ± 0.08 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d4096 | 2562.47 ± 21.71 | 909.77 ± 6.75 | 799.29 ± 6.75 | 909.81 ± 6.75 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d4096 | 33.41 ± 0.05 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d8192 | 2832.52 ± 12.34 | 3002.66 ± 12.57 | 2892.18 ± 12.57 | 3002.70 ± 12.57 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d8192 | 31.38 ± 0.06 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d8192 | 2261.83 ± 10.69 | 1015.96 ± 4.29 | 905.48 ± 4.29 | 1016.00 ± 4.29 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d8192 | 30.55 ± 0.08 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d16384 | 2473.70 ± 2.15 | 6733.76 ± 5.76 | 6623.28 ± 5.76 | 6733.80 ± 5.75 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d16384 | 27.89 ± 0.04 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d16384 | 1824.55 ± 6.32 | 1232.96 ± 3.89 | 1122.48 ± 3.89 | 1233.00 ± 3.89 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d16384 | 27.21 ± 0.04 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d32768 | 2011.11 ± 2.40 | 16403.98 ± 19.43 | 16293.50 ± 19.43 | 16404.03 ± 19.43 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d32768 | 22.09 ± 0.07 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d32768 | 1323.21 ± 4.62 | 1658.25 ± 5.41 | 1547.77 ± 5.41 | 1658.29 ± 5.41 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d32768 | 21.81 ± 0.07 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d65535 | 1457.71 ± 0.26 | 45067.98 ± 7.94 | 44957.50 ± 7.94 | 45068.01 ± 7.94 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d65535 | 15.72 ± 0.04 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d65535 | 840.36 ± 2.35 | 2547.54 ± 6.79 | 2437.06 ± 6.79 | 2547.60 ± 6.80 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d65535 | 15.63 ± 0.02 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d100000 | 1130.05 ± 1.89 | 88602.31 ± 148.70 | 88491.83 ± 148.70 | 88602.37 ± 148.70 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d100000 | 12.14 ± 0.02 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d100000 | 611.01 ± 2.50 | 3462.39 ± 13.73 | 3351.90 ± 13.73 | 3462.42 ± 13.73 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d100000 | 12.05 ± 0.03 |
llama-benchy (build: 30c3d5d)
date: 2026-01-06 07:49:53 | latency mode: generation
How to interpret data:
ctx_pp @ d32768 shows context prefill 32768 tokens long.
Which is then followed by pp2048 @ d32768 which means that we send a new request with the same context, but add a prompt 2048 tokens long and measure the speed of processing these 2048 tokens only, since the context is cached by that time.
Same with tg values. In this case we generate 32 tokens and measure generation speed.
ttfr is time from request until the first data chunk (even if it’s empty) - usually signals the end of prefill stage.
est_ppt is the estimated time of actual prompt processing, accounting for the server overhead that the tool tries to estimate in the beginning.
e2e_ttft is the time between sending the request and getting the first actual token back.
-
Ha! to the tool’s name. Well chosen.
-
Thank you. Those speeds are in the sweet spot while remaining in-loop. Now to convince my wife we need a second one…
LOL :)
I convinced my wife to buy a second one…but Minimax M2.1 token generation is about 26 token/second with nearly no context. Well…I promised her something diffrent :-).
I use QuantTrio’s AWQ. Is there such a big difference to cyankiwi’s? This is my command:
./launch-cluster.sh exec vllm serve QuantTrio/MiniMax-M2.1-AWQ \
-tp 2 \
–gpu-memory-utilization 0.7 \
–max-model-len 65535 \
–distributed-executor-backend ray \
–host 0.0.0.0 \
–port 8000 \
–trust-remote-code \
–enable-expert-parallel \
–enable-auto-tool-choice \
–tool-call-parser minimax_m2 \
–reasoning-parser minimax_m2_append_think \
You can always talk to her about the fact you’re wasting a $1500 infiniband module you’ve already paid for if you don’t buy the second one. Works like a charm. Also make sure you mention the sparks are on your will.
Off-topic, but FWIW you can save a little $ on a second Spark by getting one of the partner (ASUS/HP/MSI/etc) models with 1TB storage, and then upgrade using a Corsair MP700 Micro NVMe drive (if you’re not squeamish about all the warranty warnings).
Just went through this process myself; ended up saving about $700.
Yes. Usually I prefer QuantTrio quants, but this time cyankiwi is a better one. Cyankiwi also lists a calibration dataset on the model page and perplexity test results.
| model | test | t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|
| QuantTrio/MiniMax-M2.1-AWQ | pp2048 | 3353.83 ± 7.39 | 710.53 ± 1.35 | 610.65 ± 1.35 | 710.57 ± 1.35 |
| QuantTrio/MiniMax-M2.1-AWQ | tg32 | 25.51 ± 0.16 | |||
| QuantTrio/MiniMax-M2.1-AWQ | ctx_pp @ d4096 | 3015.02 ± 8.61 | 1458.42 ± 3.88 | 1358.54 ± 3.88 | 1458.47 ± 3.88 |
| QuantTrio/MiniMax-M2.1-AWQ | ctx_tg @ d4096 | 24.81 ± 0.03 | |||
| QuantTrio/MiniMax-M2.1-AWQ | pp2048 @ d4096 | 2469.05 ± 10.58 | 929.36 ± 3.55 | 829.48 ± 3.55 | 929.40 ± 3.55 |
| QuantTrio/MiniMax-M2.1-AWQ | tg32 @ d4096 | 24.17 ± 0.05 | |||
| QuantTrio/MiniMax-M2.1-AWQ | ctx_pp @ d8192 | 2722.02 ± 5.02 | 3109.30 ± 5.41 | 3009.42 ± 5.41 | 3109.34 ± 5.41 |
| QuantTrio/MiniMax-M2.1-AWQ | ctx_tg @ d8192 | 22.76 ± 0.58 | |||
| QuantTrio/MiniMax-M2.1-AWQ | pp2048 @ d8192 | 2176.73 ± 8.77 | 1040.76 ± 3.80 | 940.88 ± 3.80 | 1040.82 ± 3.78 |
| QuantTrio/MiniMax-M2.1-AWQ | tg32 @ d8192 | 22.69 ± 0.08 | |||
| QuantTrio/MiniMax-M2.1-AWQ | ctx_pp @ d16384 | 2349.11 ± 50.57 | 7077.72 ± 152.54 | 6977.84 ± 152.54 | 7077.76 ± 152.53 |
| QuantTrio/MiniMax-M2.1-AWQ | ctx_tg @ d16384 | 20.91 ± 0.05 | |||
| QuantTrio/MiniMax-M2.1-AWQ | pp2048 @ d16384 | 1763.82 ± 6.94 | 1261.01 ± 4.56 | 1161.13 ± 4.56 | 1261.06 ± 4.56 |
| QuantTrio/MiniMax-M2.1-AWQ | tg32 @ d16384 | 20.55 ± 0.01 | |||
| QuantTrio/MiniMax-M2.1-AWQ | ctx_pp @ d32768 | 1916.63 ± 13.68 | 17197.41 ± 121.69 | 17097.53 ± 121.69 | 17197.45 ± 121.69 |
| QuantTrio/MiniMax-M2.1-AWQ | ctx_tg @ d32768 | 17.58 ± 0.05 | |||
| QuantTrio/MiniMax-M2.1-AWQ | pp2048 @ d32768 | 1275.76 ± 5.12 | 1705.23 ± 6.46 | 1605.35 ± 6.46 | 1705.31 ± 6.45 |
| QuantTrio/MiniMax-M2.1-AWQ | tg32 @ d32768 | 17.32 ± 0.03 |
llama-benchy (build: 30c3d5d)
date: 2026-01-05 11:53:36 | latency mode: generation
| model | test | t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 | 3174.38 ± 501.64 | 772.54 ± 117.99 | 663.81 ± 117.99 | 772.59 ± 117.98 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 | 35.30 ± 1.74 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d4096 | 3165.86 ± 11.32 | 1402.55 ± 4.62 | 1293.82 ± 4.62 | 1402.58 ± 4.62 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d4096 | 34.88 ± 0.05 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d4096 | 2597.19 ± 13.33 | 897.30 ± 4.03 | 788.56 ± 4.03 | 897.34 ± 4.04 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d4096 | 33.91 ± 0.02 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d8192 | 2862.02 ± 2.29 | 2971.05 ± 2.29 | 2862.32 ± 2.29 | 2971.08 ± 2.29 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d8192 | 31.89 ± 0.03 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d8192 | 2271.24 ± 13.38 | 1010.48 ± 5.32 | 901.74 ± 5.32 | 1010.51 ± 5.31 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d8192 | 31.05 ± 0.10 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d16384 | 2477.06 ± 3.23 | 6723.05 ± 8.62 | 6614.31 ± 8.62 | 6723.08 ± 8.62 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d16384 | 27.85 ± 0.08 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d16384 | 1810.17 ± 4.68 | 1240.13 ± 2.92 | 1131.39 ± 2.92 | 1240.18 ± 2.93 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d16384 | 27.33 ± 0.05 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d32768 | 2011.29 ± 5.84 | 16400.92 ± 47.44 | 16292.19 ± 47.44 | 16400.97 ± 47.45 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d32768 | 22.17 ± 0.02 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d32768 | 1313.53 ± 3.95 | 1667.90 ± 4.70 | 1559.17 ± 4.70 | 1667.93 ± 4.70 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d32768 | 21.87 ± 0.03 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d65535 | 1457.45 ± 1.14 | 45074.37 ± 35.34 | 44965.64 ± 35.34 | 45074.41 ± 35.34 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d65535 | 15.77 ± 0.06 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d65535 | 839.79 ± 2.19 | 2547.46 ± 6.37 | 2438.72 ± 6.37 | 2547.51 ± 6.39 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d65535 | 15.67 ± 0.04 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_pp @ d100000 | 1126.08 ± 1.25 | 88912.49 ± 98.78 | 88803.75 ± 98.78 | 88912.55 ± 98.76 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx_tg @ d100000 | 12.18 ± 0.01 | |||
| cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d100000 | 606.20 ± 0.84 | 3487.19 ± 4.69 | 3378.45 ± 4.69 | 3487.27 ± 4.69 |
| cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d100000 | 12.06 ± 0.02 |
llama-benchy (0.1.1.dev1+g7646c3141)
date: 2026-01-08 18:14:28 | latency mode: generation
Since this post is pretty active, I’ll briefly explain the actual hesitation for a second unit. Note, this is all observational as benchmarking is likely unnecessary for now, but my original goal was to get agentic tasks up and running unattended for some length of time to help me with side projects.
With Opencode + llama.cpp I noticed that after a few compactions the TPS on prefill/generation plummets, the model’s output quality drops (sometimes into loops), tool calls start to fail, and the model/container needs restarting. So I trialed the eugr vllm build where I could. The concurrent performance is a great bump but I also noticed some tool call failures and loss in output quality after compactions.
Here’s where I don’t know what I’m talking about so I’ll paraphrase M2.1’s summary - CUDA graph state corruption during KV cache clearing. This is making it to where I have to babysit the session, defeating the purpose of having a local device slowly churn through work while I’m focused elsewhere (kids, fitness, chores, projects, etc).
Do others experience this?
I haven’t played with opencode yet, but does it support sub-agents?
A workflow using sub-agents may help with the context management and reduce the need for compaction.
I would envision it as:
- an orchestrator agent “plans” the task
- the orchestrator provides discrete tasks to sub-agents
- periodic reminders happen to the orchestrator of its todo list
For sure. Agents pretty much do that out of the gate. To start simply I began with basic agent profiles while omitting subagents but am currently using GPT to best implement the tactics from both the Confucius Meta/Harvard paper + Boris Cherny techniques into a new agentic scaffold. Still framing this so don’t have any experience running it yet (if it does at all).
However, some tasks just take tokens. For instance, MS’s markitdown MCP server is pretty trash. It’s good enough for a first-pass of a webpage, but using it on any technical doc/journal/patent is useless. The obvious is to setup an OCR pipeline with a vision model. GLM-4.6V does a good job at recreating markdown, but you can realistically only process ~15 pages before compaction. Likewise, prepping these converted pages into RAG-ready knowledge blocks with metadata becomes another token-hitter.
Depends on the model and setup. I haven’t experienced outright failures with M 2.1 yet up to 120K context, but that quality drop happens even with Claude and Gemini. In many cases it’s not even a malfunction, but rather a side effect of multi-turn conversations - so called “context poisoning”. It is especially pronounced when you have to correct model output somewhere in the middle.
I keep trying time after time, but in pretty much every case I come back to just using LLMs for incremental changes. Even when bootstrapping the project, I limit the initial functionality so the model doesn’t get overwhelmed.
Some coding tools manage this better than others. So far I’ve had the most success with Roo Code (or Cline when they break something in Roo again) and VS Code Insiders Edition. Claude Code was hit and miss even with Claude.
Right, and perhaps this is a pointless circle to chase, but that’s also how I have to use LLMs and what I also want to improve. Holding onto hope that a well crafted, bare bones scaffold will work.
The only push back is after watching failures with GLM-4.6V without me interjecting comments/corrections. This particular prompt couldn’t be any more straight forward and well defined. Yet after a few compactions, “my instructions say to process but I’ll ask the user if I should process the next page” repeatedly. You know the feeling of turning on GPT-OSS’s high-thinking or anything Qwen3? Basically gets there on its own.