MiniMax-M3-W4A16-GPTQ 2xGB10 Deployment (36 t/s) (fp8, nvfp4, KVarN, EAGLE-3)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

TLDR - Quantized and calibrated MiniMax-M3-GPTQ checkpoint paired with 2xGB10 deployment package. Optimized fp8, nvfp4, KVarN, and EAGLE-3 verify kernels integrated with b12x and vllm. Vision is not tested or validated but should work in theory.

Container, build steps, recipes, and checkpoints:

This package is not stable. All three recipes push the absolute limit of the available system memory. I run my nodes headless and clear the page caches before each cluster launch. Expect, errors, OOMs, etc before you find the sweet spot for your configuration. Bring your agent.

Credit to @PILCOTHINK for both introducing KVarN as a potential optimization and validating the implementation directly: Serving Qwen3.5-397B-A17B at 1M Tokens on 2× DGX Spark — MiniMax M3 Is Next

His calculations for KV-cache size potential were spot on.

fp8 w/EAGLE-3 - 131k ctx
nvfp4 w/EAGLE-3 - 196k ctx
KVaRN - 262k ctx, up to 370k ctx observed

I spent a lot of time producing the quantization checkpoint. The calibration dataset included simulated agentic trajectories using the MiniMax-M3 api rendered through pi and OpenCode for SWE and terminal tasks. Math, retail, and general chat were also included domains. With that said, I cannot make any concrete claims to actual quality of the quant - what I do know is that I did my due diligence, for what its worth, to do the GPTQ properly without cutting corners. How well this worked, or whether it worked at all, I don’t know. On tool-eval-bench hardmode in a best of 5 with fp8 KV-cache, the quant scored on par with the MiniMax-M3 api. Admittedly, both the quant and the api had mediocre scores, averaging around 77/100.

I recommend using adaptive thinking (included in the recipe kwargs, along with important reasoning parser/chat template fixes in the mods). Throughput benchmarks are below for nvfp4-EAGLE3 and KVaRN. fp8-EAGLE-3 has comparable throughput to nvfp4-EAGLE-3. Prompt processing is low due to lowering max_num_batched_tokens to 1024 to increase KV-cache size. But more aggressive batching is theoretically possible if that is priority for your workload. Adding the EAGLE-3 drafter to the KVarN configuration hurts available KV-cache more than the other KV cache quants. So the recipe leaves it out, but EAGLE-3 is still compatible with it. Something else worth noting is that the EAGLE-3 drafter results in diminishing returns at longer context lengths. My hypothesis for this is that the target verify step is more costly, because the target model needs to produce top-k block scores for a longer context length for each drafted token, hence hurting throughput at longer context lengths.

nvfp4-EAGLE-3

| model                         |            test |             t/s |     peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:------------------------------|----------------:|----------------:|-------------:|-----------------:|-----------------:|-----------------:|
| Sebesky/MiniMax-M3-W4A16-GPTQ |          pp2048 | 1376.00 ± 12.12 |              |  2080.75 ± 13.04 |  1488.49 ± 13.04 |  2145.45 ± 14.66 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |            tg32 |    35.49 ± 4.95 | 36.79 ± 4.96 |                  |                  |                  |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  ctx_pp @ d4096 |  1182.70 ± 2.50 |              |   4056.37 ± 7.35 |   3464.12 ± 7.35 |   4119.00 ± 3.24 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  ctx_tg @ d4096 |    25.85 ± 2.23 | 27.33 ± 2.36 |                  |                  |                  |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  pp2048 @ d4096 |  964.46 ± 13.72 |              |  2716.15 ± 29.93 |  2123.89 ± 29.93 |  2781.90 ± 30.29 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |    tg32 @ d4096 |    31.87 ± 1.12 | 32.93 ± 1.16 |                  |                  |                  |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  ctx_pp @ d8192 |  1076.15 ± 3.53 |              |  8205.60 ± 24.95 |  7613.34 ± 24.95 |  8262.40 ± 19.94 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  ctx_tg @ d8192 |    27.01 ± 1.61 | 30.49 ± 2.26 |                  |                  |                  |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  pp2048 @ d8192 |  952.11 ± 17.91 |              |  2744.04 ± 40.55 |  2151.78 ± 40.55 |  2812.80 ± 41.01 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |    tg32 @ d8192 |    30.82 ± 3.51 | 31.59 ± 3.76 |                  |                  |                  |
| Sebesky/MiniMax-M3-W4A16-GPTQ | ctx_pp @ d16384 |  1016.11 ± 0.79 |              | 16717.49 ± 12.49 | 16125.23 ± 12.49 | 16772.56 ± 12.27 |
| Sebesky/MiniMax-M3-W4A16-GPTQ | ctx_tg @ d16384 |    29.25 ± 1.88 | 30.72 ± 4.18 |                  |                  |                  |
| Sebesky/MiniMax-M3-W4A16-GPTQ | pp2048 @ d16384 |   886.72 ± 7.38 |              |  2902.05 ± 19.33 |  2309.79 ± 19.33 |  2959.50 ± 16.64 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |   tg32 @ d16384 |    32.84 ± 2.91 | 33.70 ± 3.33 |                  |                  |                  |
| Sebesky/MiniMax-M3-W4A16-GPTQ | ctx_pp @ d32768 |   959.78 ± 2.51 |              | 34734.21 ± 89.70 | 34141.95 ± 89.70 | 34791.61 ± 88.23 |
| Sebesky/MiniMax-M3-W4A16-GPTQ | ctx_tg @ d32768 |    27.87 ± 1.92 | 30.61 ± 3.01 |                  |                  |                  |
| Sebesky/MiniMax-M3-W4A16-GPTQ | pp2048 @ d32768 |   782.12 ± 3.49 |              |  3210.84 ± 11.66 |  2618.58 ± 11.66 |  3270.24 ± 18.36 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |   tg32 @ d32768 |    24.35 ± 2.42 | 26.67 ± 2.62 |                  |                  |                  |

KVarN

| model                         |            test |             t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:------------------------------|----------------:|----------------:|-------------:|------------------:|------------------:|------------------:|                            | Sebesky/MiniMax-M3-W4A16-GPTQ |          pp2048 | 1767.04 ± 48.35 |              |   1439.90 ± 32.36 |   1159.88 ± 32.36 |   1483.13 ± 13.15 |                            | Sebesky/MiniMax-M3-W4A16-GPTQ |            tg32 |    22.65 ± 0.01 | 23.00 ± 0.00 |                   |                   |                   |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  ctx_pp @ d4096 |  1332.75 ± 1.34 |              |    3353.85 ± 2.86 |    3073.84 ± 2.86 |    3419.90 ± 4.94 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  ctx_tg @ d4096 |    22.60 ± 0.02 | 23.00 ± 0.00 |                   |                   |                   |                            | Sebesky/MiniMax-M3-W4A16-GPTQ |  pp2048 @ d4096 |   937.57 ± 8.59 |              |   2464.57 ± 20.15 |   2184.55 ± 20.15 |    2510.73 ± 7.26 |                            | Sebesky/MiniMax-M3-W4A16-GPTQ |    tg32 @ d4096 |    22.39 ± 0.19 | 23.00 ± 0.00 |                   |                   |                   |
| Sebesky/MiniMax-M3-W4A16-GPTQ |  ctx_pp @ d8192 |  1147.31 ± 7.91 |              |   7421.13 ± 48.78 |   7141.11 ± 48.78 |   7488.76 ± 52.95 |                            | Sebesky/MiniMax-M3-W4A16-GPTQ |  ctx_tg @ d8192 |    22.30 ± 0.06 | 23.00 ± 0.00 |                   |                   |                   |                            | Sebesky/MiniMax-M3-W4A16-GPTQ |  pp2048 @ d8192 |   913.49 ± 9.47 |              |   2522.22 ± 23.41 |   2242.20 ± 23.41 |   2568.48 ± 31.48 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |    tg32 @ d8192 |    22.24 ± 0.24 | 23.00 ± 0.00 |                   |                   |                   |                            | Sebesky/MiniMax-M3-W4A16-GPTQ | ctx_pp @ d16384 |  1053.77 ± 6.43 |              |  15829.54 ± 95.34 |  15549.53 ± 95.34 |  15896.51 ± 94.64 |                            | Sebesky/MiniMax-M3-W4A16-GPTQ | ctx_tg @ d16384 |    22.15 ± 0.01 | 23.00 ± 0.00 |                   |                   |                   |                            | Sebesky/MiniMax-M3-W4A16-GPTQ | pp2048 @ d16384 |   890.17 ± 4.65 |              |   2580.77 ± 12.02 |   2300.75 ± 12.02 |   2642.97 ± 13.78 |
| Sebesky/MiniMax-M3-W4A16-GPTQ |   tg32 @ d16384 |    22.21 ± 0.01 | 23.00 ± 0.00 |                   |                   |                   |                            | Sebesky/MiniMax-M3-W4A16-GPTQ | ctx_pp @ d32768 |  1001.66 ± 4.03 |              | 32995.13 ± 131.94 | 32715.11 ± 131.94 | 33045.46 ± 114.71 |                            | Sebesky/MiniMax-M3-W4A16-GPTQ | ctx_tg @ d32768 |    21.78 ± 0.03 | 22.00 ± 0.00 |                   |                   |                   |
| Sebesky/MiniMax-M3-W4A16-GPTQ | pp2048 @ d32768 |   832.69 ± 3.10 |              |    2739.56 ± 9.15 |    2459.54 ± 9.15 |    2786.99 ± 3.53 |

Hoping this serves as a baseline for more experimentation, or even real workloads on the 2xGB10.

2

Well done! This is a big progress!

No matter what some users with high hallucinations will say 😅 lol

3

Wow, I didn’t think full W4A16 would be viable on 2 GB10s and figured the mixed 3.2 bit Autoround would be the best option for this combination.

Very cool. Will check it out and compare with the 3.2 bit.

4

Amazing! I tired this on my 4 node cluster, dropping kvarn and using an NVFP4 quant. I get a flat 40 tok/s decode throughout ctx depth and ~1200 tok/s prefill. Before this M3 was DOA for me, but now I want to test it to see if it can be a daily driver and make GLM feature dev only.

5

Nice! Are you using the recipe defaults? With 4 nodes I’m thinking you might be able to increase max_num_batched tokens for better prefill potentially

6

I changed it to 8192, but the result was within noise. Maybe it’s a b12x issue. I had a dramatic drop in prefill with it and DSV4 Flash too a while ago.

7

Hi @a3refaat, thank you for sharing this work.

I have a similar 2x GB10 setup and am considering the NVFP4 + EAGLE-3 recipe.

Could this setup realistically serve around a 100k-token context in normal use? If you have tested near 100k context, could you share the approximate real-world numbers for:

  • prefill throughput
  • TTFT / TTFR
  • decode speed (tok/s)
  • aggregate throughput under batching

Also, how many concurrent sequences / requests can it reasonably handle at around 100k context?

More specifically, what max_num_seqs and max_num_batched_tokens would you consider practical and stable for a 100k-context configuration on 2x GB10?

Thanks!

8

You’re asking for 100k — on the same 2× GB10, we’re serving MiniMax-M3 at 229k. Let me just hand you the recipe.

We run Sebesky/MiniMax-M3-W4A16-GPTQ at max_model_len = 229,376 across two DGX Sparks (TP=2), with EAGLE-3 speculative decoding — the exact path you’re after. Draft model is MiniMax-M3-EAGLE3-RTN-INT4 at draft_TP=2.

The recipe (vLLM, 2× GB10, one GPU per node):

model: Sebesky/MiniMax-M3-W4A16-GPTQ

--tensor-parallel-size 2

--max-model-len 229376

--max-num-batched-tokens 1024

--max-num-seqs 1 # single-stream here; raise it for concurrency

--speculative-config '{“model”:“…/MiniMax-M3-EAGLE3-RTN-INT4”,

                 "draft_tensor_parallel_size":2,

                   "num_speculative_tokens":3,

                   "attention_backend":"b12x"}'

Where we actually sit:

Context: 229k — that’s 2.3× the 100k you’re asking for, live and stable.

~40+ tok/s overall, TTFT ~1.5s. That’s EAGLE-3 earning its keep — it’s what makes a model this size genuinely usable at this context on two small boxes.

For your 100k: you’re comfortably under our ceiling, so it’s the easy case. Set --max-model-len 131072; since 100k costs less KV per sequence, you can push --max-num-seqs up for real concurrency (we hold it at 1 because M3 backs an agent).

Two scars so you skip them: EAGLE-3 draft is happiest at num_speculative_tokens=3 — higher got unstable for us. And boot errors about “drafting layers / kv cache group” are almost always image/runtime drift, not your config — rebuild the overlay from the right commit.

Bottom line: 100k isn’t the wall on 2× GB10 — we’re past 2× that today, at ~40+ tok/s, with EAGLE-3. Recipe’s above; happy to trade the full compose/env.

I heard that Mia is disappointed with M3, because she said it was acting out on its own, but I haven’t seen that behavior. As far as I’m concerned, this is API quality and speed, close to bulletproof, and I’d put it up against Claude and Deepseek any day.

9

Would be lovely to see tool-eval-bench --hardmode output on 84 question test when you get a time to run it - compare back to back with deepseek v4 flash dspark and mimi v2.5 dspark

10

Thanks for sharing the recipe — the EAGLE-3 / TP=2 setup is very close to what I am testing. I would separate the configured context ceiling from long-prefix TTFT, though.

On my current two-GB10 M3 endpoint, the live API reports max_model_len=65,536, with TP=2, EAGLE-3, NVFP4 KV, max_num_batched_tokens=8192, max_num_seqs=2, chunked prefill, and prefix caching enabled.

With a real 59,501-token cache-miss prompt, I measure 43.714 s to first streamed token, or about 1,361 prompt tok/s end-to-end. Repeating the exact same prefix immediately gives a 99.60% prefix-cache hit and 1.384 s TTFT.

So I can reproduce a ~1.x-second first token only in the cache-hit case, not when a new ~60K document has to be prefetched.

My question is therefore about the measurement conditions behind the 229K / ~1.5 s number: actual prompt-token count, cold cache vs prefix-cache hit, warm-up/JIT state, output length, and whether TTFT means first streamed content token.

If that 1.5 s is for a real 100K–229K cache miss, I would love to see the /metrics cache size and the benchmark command, because that would be materially better than what I see on the same class of hardware. If it is a warm cached or short-prompt TTFT, that is still useful, but it is a different metric.

For my workload, M3 works well as a local long-source and privacy-preserving extraction engine, while a hosted DeepSeek V4 Flash run was much faster and more complete for a 76-minute transcript plus six structured deliverables. I am therefore measuring prefill, decode, cache hit rate, TTFT, output completion, and task quality separately rather than treating one headline tok/s number as the whole story.

11

Clarifying the measurement conditions.

Setup: MiniMax-M3 (428B) on 2× DGX Spark GB10, TP=2, EAGLE-3, nvfp4 KV, chunked prefill, prefix caching, max_num_batched_tokens=1024, max_num_seqs=1, max_model_len=229376.

The ~1.5 s figure was the warm prefix-cache / short-prompt TTFT, not a cold large-prefix miss. Cold picture, measured on the live endpoint — fresh unique prompts, 0% prefix hit, output capped at 2 tokens, TTFT = first streamed content token:

50,279 tokens → 52.9 s to first token (~950 tok/s prefill)
99,929 tokens → 123.1 s (~812 tok/s)
149,468 tokens → 214.0 s (~698 tok/s)
199,663 tokens → 317.0 s (~630 tok/s)
~240,000 tokens → rejected at max_model_len=229376

A ~200K cold prefill completes in ~5.3 min. Prefill is bandwidth-bound on the GB10 (~273 GB/s) and slows with length as attention grows. Warm, the same prefix returns in ~1 s at a 99.7% cache hit. Different metrics, both real.

The ceiling is enforced: a ~240K request is rejected at max_model_len=229376; a 199,663-token prompt prefills and returns cleanly. So 224K is a real, usable window, not a paper number — and the model holds coherence across it.

On 1024 vs 8192: at 8192 the larger prefill buffer reduces available KV to 5.15 GiB against the 5.55 GiB needed for the 229K ceiling, and the engine won’t start (estimated max len ~212K). At 64K context there’s KV headroom to spend on prefill; at 224K there isn’t. It’s a context-vs-prefill-throughput allocation on fixed memory. EAGLE-3 accelerates decode only, so it doesn’t affect the prefill/TTFT figures above.

On DeepSeek V4-Flash: Published quality is close and slightly favors M3 (Artificial Analysis Intelligence Index 44 vs 40; M3 ahead on Terminal-Bench and SWE-bench-Pro). For local, privacy-preserving long-source extraction where the material never leaves the machine, M3 at 224K on two Sparks has been excellent for our workload. For heavier throughput, a second Spark pair to run the larger DeepSeek locally is the next step.

Prefill / decode / cache-hit / TTFT / output-completion / quality best measured as separate axes. /metrics prefix-cache counters and the exact benchmark harness are available for any rung above.

One tuning note specific to M3: the model is commonly served with chat_template_kwargs.thinking_mode=“adaptive” as the default, and in long agentic / tool-calling loops that setting is the source of the “runaway” behavior reported elsewhere — under adaptive, the model decides per-turn whether to emit a reasoning block, and it increasingly elects not to, so after a few tool results it starts acting without deliberation: drifting off task, skipping instructions, and taking steps on its own. The fix is to pin thinking_mode=“enabled” (via chat_template_kwargs / extra_body) rather than leaving it adaptive; enabled forces a step-by-step reasoning pass before every response, including after each function/tool result, which is exactly the discipline an agent needs to stay grounded across a multi-step loop. In practice the cost is a small, largely unnoticeable latency bump, and the gain is a marked jump in lucidity and instruction-following — so for any agentic or tool-using deployment, M3 should be run in explicit thinking mode, not adaptive. Adaptive is fine for one-shot chat; it is the wrong default the moment tools and multi-turn autonomy are involved.

12

What image are you using? Spark-vllm-docker or something else custom? I’m very impressed with DS4-Flash on 2 Sparks but would like to look at this. Your recipe is almost reproducible save the actual image…

13

It’s the community harness (credit to eugr). It builds the container rather than shipping a prebuilt one, but the heavy lifting is done for you: it pulls prebuilt vLLM + FlashInfer wheels from GitHub releases, so a build is ~2–3 min after the base image pull, no compiling from source unless you ask for it (–rebuild-vllm). That’s the “missing image” — you generate it locally in a couple minutes.It has a dedicated MiniMax-M3 deployment package (see DEPLOYMENT.md in the repo). The exact recipe I’m running is minimax-m3.

git clone GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub

cd spark-vllm-docker

# 2. connect the two Sparks (ConnectX/RoCE link + passwordless SSH head->worker),

# then generate the cluster .env (HEAD_IP / PEER_NODES / COPY_HOSTS):

./autodiscover.sh # see docs/NETWORKING.md

# 3. build the image and distribute it across the cluster (~2-3 min)

./build-deploy.sh # M3 package build; or ./build-and-copy.sh -c for the generic build

# 4. HF token, then download + fan the weights out over the interconnect

export HF_TOKEN=…

./hf-download.sh Sebesky/MiniMax-M3-W4A16-GPTQ -c --copy-parallel

# 5. launch the recipe (applies the b12x/nvfp4 + EAGLE-3 mods into a fresh container)

./run-recipe.sh --no-ray --gpu-mem 0.94 \

-d recipes/minimax-m3-w4a16-gptq-b12x-nvfp4-eagle3.yaml

That serves Sebesky/MiniMax-M3-W4A16-GPTQ on :8000 across both nodes, with the reasoning parser + tool-calling (minimax_m3) and the EAGLE-3 drafter baked in by the recipe — no manual vLLM patching. The mods are applied into the container at launch, which is how it keeps the mutually-exclusive attention stacks from colliding. DEPLOYMENT.md documents all three KV variants (fp8 / nvfp4 / KVarN) if you want to trade context-per-byte vs decode speed.

-tp 2

--max-model-len 229376

--gpu-memory-utilization 0.94

--max-num-batched-tokens 1024

--max-num-seqs 1

--block-size 128

--kv-cache-dtype nvfp4

--attention-backend b12x

--compilation-config ‘{“mode”:0,“cudagraph_mode”:“FULL_DECODE_ONLY”,“cudagraph_copy_inputs”:true}’

--speculative-config ‘{“method”:“eagle3”,“model”:“Sebesky/MiniMax-M3-EAGLE3-RTN-INT4”,“draft_tensor_parallel_size”:2,“num_speculative_tokens”:3,“attention_backend”:“b12x”}’

--reasoning-parser minimax_m3 --tool-call-parser minimax_m3 --enable-auto-tool-choice

--default-chat-template-kwargs ‘{“thinking_mode”:“adaptive”}’ (this is default, change to ‘enabled’ here or at the harness level)

14

I do have a question for this, the context is definetly nice but is this WITH the vision ?

15

Vision is definitely not activated. I’m using MiMo-VL-7B on a separate 1080 ti, which also ingests video.

Activating vision on this module would eat ~198K tokens of context — about 86% of the entire window. M3 would drop from 229K → roughly ~30K tokens (the ~0.75 GiB of KV left after the vision buffer).

And that’s before images. Every image then spends hundreds to a few thousand more tokens out of that remaining ~30K, plus its own KV — so in practice the usable text window with vision on would be well under 30K.

16

Ok it makes sense now. I have a similar repo with 3 choices GitHub - tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s: MiniMax-M3 (428B, no pruning) at 36 tok/s on 2× NVIDIA DGX Spark — W4A16 GPTQ + NVFP4 KV + EAGLE-3 speculative decoding on vLLM. Three serving lanes: speed / balanced / long-context. · GitHub

lane config our decode t/s (2026-07-05) author’s decode t/s max ctx / KV pool status
A — BALANCED b12x nvfp4 KV + EAGLE3 spec-3, GMU 0.929 36.6 JSON / 31.8 code (thinking disabled) tg32 35.49, peak 36.79 196K / 208,128 tokens ✅ verified, serving now
C — SPEED KVarN k4v2 + EAGLE3 ⏳ currently testing 39.7 / 30.4 / 26.4 @512/65K/120K (accept len 3.36, 78.6% @120K) 131K / 170,752 ⏳ testing on our pair
B — CONTEXT KVarN k4v2, drafterless ⏳ currently testing ~22.7, flat with depth 262K (370K observed) ⏳ testing on our pair
17

Hrmmm I can’t see to get this running on my GX10s mine keeps dying in Kernel JIT

When did you last build the image from scratch, and does a fresh build-deploy.sh still succeed today? vendor/b12x/pyproject.toml pins nvidia-cutlass-dsl>=4.5.2 with no upper bound. Our theory: your working image predates a cutlass release that breaks the MLIR bindings, so any new build now fails — nothing to do with our hardware. Your gotchas index has no cutlass entry, which suggests you never saw this.

Exact nvidia-cutlass-dsl / -libs-base / -libs-cu13 versions in the working image? We resolve to 4.5.2 across all three.

Does mods/exp-b12x’s 4.4.2 pin apply to the vendored CuTe-DSL b12x or only the flashinfer-b12x MoE path? It contradicts vendor/b12x’s >=4.5.2; 4.4.2 fails for us with no attribute 'OperandMajorMode'.

18

Exact versions in the working image (live pip list from the production container, up 4 days):

nvidia-cutlass-dsl 4.5.2

nvidia-cutlass-dsl-libs-base 4.5.2

nvidia-cutlass-dsl-libs-cu13 4.5.2

flashinfer-python 0.6.14

flashinfer-cubin 0.6.14

flashinfer-jit-cache 0.6.14

vllm 0.22.1rc1.dev535+g979b56a66.d20260706.cu132

So I have to falsify your cutlass theory on the version axis: we resolve to 4.5.2 across all three, same as you — and ours runs. Whatever is killing your kernel JIT, it isn’t 4.5.2-vs-something-older, and that’s also why there’s no cutlass entry in the gotchas index: we genuinely never hit one.

Build date, honestly: the image was last built from scratch 2026-07-06 (harness checkout f942437, Jul 4). I have not re-run build-deploy.sh since, so I can’t certify a from-scratch build succeeds today — that part of your theory stands untested. But note where the real drift risk is: the harness pulls prebuilt vLLM + FlashInfer wheels from the GH releases at build time. Our wheel is datestamped d20260706. A fresh build today pulls whatever’s newest — so if new builds break with identical cutlass versions, my money is on a newer vLLM/FlashInfer wheel regression, not cutlass. Suggestion: pin the harness at f942437 and install the same d20260706 wheel set, and I’d expect your JIT failure to move or vanish.

On the 4.4.2 pin — it does not apply to the vendored b12x path, and here’s the proof: our recipe (minimax-m3-w4a16-gptq-b12x-nvfp4-eagle3.yaml) applies exactly one mod group, mods/minimax-m3-gptq-b12x-eagle3. Nothing in our launch path references mods/exp-b12x — that’s a separate experimental mod (the flashinfer-b12x MoE path), and its run.sh is where the 4.4.2 pin lives. Empirical confirmation: the running production container shows 4.5.2, i.e. that pin never executed in this deployment. Your no attribute ‘OperandMajorMode’ on 4.4.2 is exactly what I’d expect — the vendored b12x CuTe-DSL uses ≥4.5.2 API surface. Don’t combine exp-b12x with the vendored path; the pins genuinely contradict because they belong to different stacks.

One more thing worth checking for a “dies in kernel JIT” specifically: in our image all three flashinfer packages (-python, -cubin, -jit-cache) are version-locked at 0.6.14 — the prebuilt jit-cache wheel means almost nothing JIT-compiles at boot. If your build ended up with a missing or mismatched flashinfer-jit-cache/flashinfer-cubin, FlashInfer falls back to compiling kernels at runtime, which on a GB10 at --gpu-memory-utilization 0.94 is a good way to die mid-JIT. Check those three are present and aligned before anything else. If it still dies, post the actual JIT error text and I’ll diff it against our boot logs.

19

thanks looks like we pulled a new version of flashinfer

flashinfer-jit-cache 0.6.15 vs your flashinfer-jit-cache 0.6.14

and that would definitely break the JIT compile thanks so much rebuilding now. If this doesn’t fix it I’ll take you up on the offer to the look at the JIT error text.

Thanks

20

Thanks — that falsified my cutlass theory cleanly, and your wheel-drift instinct pointed at the right class of problem even though the specific package turned out different. Full results:

Your redirect was right about the mechanism, wrong about which wheel. I pinned FlashInfer to a verified-aligned 0.6.13 trio (installed from local wheels, versions confirmed before/after — my first attempt silently no-op’d because uv pip install without --system doesn’t touch dist-packages, so ignore any earlier claim I made there). Result: byte-identical failure. Two different aligned FlashInfer trios, same error. So FlashInfer version isn’t the differentiator either.

Everything is now falsified except the artifacts themselves:

  • cutlass version — identical 4.5.2 to yours
  • FlashInfer version — 0.6.15 and 0.6.13 both fail identically
  • module shadowing — retested properly; b12x.pth only adds /opt/b12x, and cutlass resolves to nvidia_cutlass_dsl/... both before and after import b12x
  • mixed/clobbered install — --force-reinstall of all three cutlass packages changes nothing
  • b12x/lane-specificity — the KVarN lane (stock Triton, zero b12x mods, no attention_backend: b12x) fails identically, so the CuTe-DSL JIT is reached via the Marlin int4 MoE path regardless of attention backend

The actual failure, verbatim:
cutlass/cutlass_dsl/cutlass.py:358: self.gpu_module = gpu.GPUModuleOp(ir.StringAttr.get(“kernels”), loc=loc)
cutlass/_mlir/dialects/_gpu_ops_gen.py:1357, in init
super().init(self.OPERATION_NAME, self._ODS_REGIONS, self._ODS_OPERAND_SEGMENTS,
self._ODS_RESULT_SEGMENTS, attributes=attributes, results=results, operands=…)
TypeError: init(): incompatible function arguments. The following argument types are supported:

  1. init(self, operation: object) → None

cutlass’s generated dialect code calls a rich OpView.__init__; its own compiled cutlass._mlir._mlir_libs._cutlass_ir._mlir.ir.OpView exposes only __init__(self, operation). Same version string as yours — different bits.

The reproducibility problem, which I think is the real story here: the harness pulls from eugr/spark-vllm-docker’s rolling tags (prebuilt-vllm-current, prebuilt-flashinfer-current). Both were republished 2026-07-19 and now carry only 0.6.15 — your 0.6.14 assets are gone. Since PyPI’s flashinfer-cubin caps at 0.6.13, flashinfer-cubin 0.6.14 is now unobtainable anywhere. And PyPI’s nvidia-cutlass-dsl 4.5.2 evidently no longer serves self-consistent artifacts. Net: your known-good image can’t be rebuilt today by anyone — including you. Worth knowing before you ever rebuild, and a dated tag (prebuilt-flashinfer-0.6.14) plus recorded wheel hashes would prevent a recurrence.

One ask: could you copy the surviving wheels out of your running container? They’re the only extant copies

Specifically flashinfer_{python,cubin,jit_cache}-0.6.14*, the nvidia_cutlass_dsl* wheels if retrievable, and the vllm ...d20260706 wheel. Failing that, a docker save of the image would let us skip the build entirely and give you a second validated deployment. Happy to host/mirror them so the next person isn’t stuck in the same hole.