TL;DR — I got the community AutoRound‑mixed quant of MiniMax‑M3 (a ~428B‑param, 128‑expert MoE with native Multi‑head Sparse Attention + a CLIP vision tower) serving across two DGX Spark (GB10) boxes with tensor‑parallel = 2 over the 200G ConnectX‑7 fabric, MSA active (not a dense fallback), and working image understanding. The quant author had only ever run it single‑node, text‑only — so the cross‑node + vision parts are new. Below is exactly what it took, including the GB10/sm_121 gotchas that will bite anyone doing MoE on these boxes.
This is a “make it work” report, not a performance brag: it’s eager‑only (~5 tok/s decode) for hardware reasons explained below. The value here is (a) it runs at all on 2× Spark, (b) the sm_121 findings, and (c) the vision‑loader trick.
Hardware / software
- 2× DGX Spark (GB10) — each 121 GB unified memory, compute capability 12.1 (sm_121). Combined ~242 GB usable, which is what makes a 189 GB model possible at all.
- Direct‑cabled ConnectX‑7 200G (RoCE); no GPUDirect on GB10, so cross‑node collectives go through host bounce buffers.
- vLLM
0.23.1rc1.devwith the native MiniMax‑M3 model (incl.MiniMaxM3SparseForConditionalGeneration+ the MSA path). - Model:
aquaman164/MiniMax-M3-AutoRound-3.2bit-longctx(188.6 GB, 36 shards). Plugin: OneCompression ( GitHub - mmzz164/OneCompression: Python package for LLM compression · GitHub , tagm3-serving-v1— Fujitsu, Apache). Credit to both.
Per‑GPU footprint at TP=2: 88.4 GiB weights/node (incl. the 3.4 GB vision tower). MSA + only 4 KV heads make KV very cheap, so a small KV pool still gives a big context window.
Step 1 — Get the quant to load (config surgery)
Unsupported weight_bits: 16— AutoRound uses a 16‑bit base with per‑module overrides; vLLM’s stock validator only accepts {2,3,4,8}. Fix: register OneComp’sautoround_mixedmethod and renamequant_method→autoround_mixedinconfig.jsonso the stock validator never fires.Unsupported activation: silu— bake the missing values intoconfig.json(top‑level andtext_config):hidden_act=swigluoai,swiglu_alpha=1.702,swiglu_beta=1.0,swiglu_limit=7.0,rope_theta=5e6,partial_rotary_factor=0.5,rotary_dim=64,head_dim=128,use_gemma_norm=true,use_qk_norm=true,qk_norm_type=per_head,dense_intermediate_size=12288,shared_intermediate_size=3072,n_shared_experts=1,scoring_func=sigmoid,use_routing_bias=true,routed_scaling_factor=2.0.n_shared_experts=1is critical — without it the shared expert is dropped in every MoE layer → garbage.
Load the plugin into vllm serve via a tiny vllm.general_plugins entry‑point shim (no fork; not pip‑installed, just on sys.path).
Step 2 — Cross‑node engine args (GB10 specifics)
--tensor-parallel-size 2 --enable-expert-parallel
--quantization autoround_mixed --trust-remote-code
--block-size 128 # MANDATORY for MSA
--attention-backend TRITON_ATTN # FLASHINFER incompatible with MSA's block_size=128
--enforce-eager # MoE cudagraph wall, see Step 3
--disable-custom-all-reduce # custom all-reduce unreliable on sm_121
--dtype bfloat16 --gpu-memory-utilization 0.85 --max-model-len 40960
Use a no‑ray multi‑node launcher — Ray’s cross‑node NCCL hangs on GB10.
Step 3 — The sm_121 wall: MoE cannot use CUDA graphs
Any cudagraph mode (incl. PIECEWISE) dies with CUDA error: illegal memory access during capture for this 128‑expert MoE. Confirmed three ways (author on sm_120, me on sm_121, and the known GB10 result: “CUDA graph capture fails for MoE dispatch patterns on sm_121 → use --enforce-eager; dense models can use cudagraphs, large MoE cannot.”). Eager is mandatory for MoE on GB10. That + per‑layer cross‑node all‑reduce over RoCE (no GPUDirect) is why decode is ~5 tok/s — structural, not the quant.
Step 4 — Vision (the part that wasn’t supposed to work)
Switch to MiniMaxM3SparseForConditionalGeneration + a custom loader. Three keys:
- The checkpoint ships vision in BOTH namings —
model.vision_tower.*(transformers) andvision_tower.vision_model.*(vLLM‑native), 515 tensors each. Skip the transformers copy, pass the vLLM‑named copy straight through the model’s ownhf_to_vllm_mapper— vision loads almost free. - Language still needs translation + de‑quant:
model.language_model.X → language_model.model.X'(expertsgate/up/down→w1/w3/w2,indexer→index_,mlp.*→block_sparse_moe.*); de‑quant the fused q/k/v (fuse with a bf16 MSA indexer) and split the pre‑fusedgate_up. - Don’t drop
lm_head— it’s a bare top‑levellm_head.weightthat must map tolanguage_model.lm_head.weight, else silent garbage logits.
Vision tower is only 3.37 GB — fits in budget. Gotcha: the first image request triggers a one‑time ~20 s flashinfer/FlashAttention ViT JIT autotune (AttentionBackendEnum.FLASH_ATTN for vit attention); use a long client timeout or a warmup image, then it’s fast.
Result
Cross‑node, TP=2, MSA active, vision working. Text: coherent with <mm:think> reasoning (“Paris”/“東京”). Vision: fed a synthetic image (red circle, blue square, green triangle, caption “M3 VISION TEST”) → M3 listed every shape, color, position, the inverted triangle, and read the text. Footprint 88.4 GiB/node, 40k context. Throughput ~5 tok/s decode / ~74 tok/s prefill — a long‑context/multimodal endpoint, not a speed daemon.
Gotcha checklist
- not doneRename
quant_method → autoround_mixed(elseweight_bits: 16reject) - not doneFORCE config in both top level and
text_config(esp.swigluoai,n_shared_experts=1) - not done
--block-size 128+--attention-backend TRITON_ATTN - not done
--enforce-eager(MoE cudagraph crashes on sm_121) - not done
--disable-custom-all-reduce - not doneVision:
ForConditionalGenerationarch, keep vLLM‑named vision weights, fix barelm_head, >150 s timeout on first image