Successfully serving MiniMax-M3-NVFP4 on 4x DGX Spark with vLLM

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

The model card for lukealonso/MiniMax-M3-NVFP4 notes that a vLLM container for SM120 is forthcoming and that the model may be difficult to run “due to the specialized kernels required.” After spending time waiting for upstream instructions I worked out a working configuration for 4x DGX Spark (GB10, sm_121) and wanted to share what worked so that other folks with DGX Spark clusters can serve the model without starting from scratch.

@eugr’s spark-vllm-docker was used for this process.

Requirements

  1. A custom container image. The model needs:

    • The chthonic vLLM base ( Package vllm-m3-chthonic · GitHub ), which bundles the ModelOpt NVFP4 runtime, the B12X MiniMax M3 sparse attention path, and the B12X NVFP4 MoE path
    • ray==2.55.1 installed (multi-node vLLM)
    • b12x from the GitHub commit that ships copy_runtime_metadata (commit 08e980c303b0b6291700a6b85aa09aa874fc27cb is what I used; without that commit signature vLLM aborts during init)
    • NCCL 2.30.7 forced from /opt/nccl230/build/lib/libnccl.so.2. The default NCCL that ships in the chthonic Python environment does not work for multi-node TP. Force it via VLLM_NCCL_SO_PATH=/opt/nccl230/build/lib/libnccl.so.2 and LD_PRELOAD on the same path, and symlink the pip-installed nvidia/nccl/lib/libnccl.so.2 to that path so any transitive loader picks it up. Verify with NCCL_DEBUG=VERSION at startup; you want NCCL version 2.30.7 in the logs.

  2. Multi-node launch must be --no-ray (PyTorch distributed), not Ray. With Ray the chthonic build does not initialize the MiniMax M3 parsers cleanly. The launcher is launch-cluster.sh --no-ray --master-port 29501.

  3. The vLLM serve flags that worked for me on 4 nodes:

    • --quantization modelopt_fp4
    • --attention-backend B12X_ATTN
    • --moe-backend b12x
    • -cc.mode=VLLM_COMPILE
    • -cc.cudagraph_mode=FULL (slightly better per-request throughput than PIECEWISE, longer warmup)
    • --load-format safetensors
    • --kv-cache-dtype auto (FP8 KV cache works but I left it off for quality)
    • --max-model-len 524288
    • --max-num-batched-tokens 8192
    • --max-num-seqs 5
    • --gpu-memory-utilization 0.85
    • --enable-chunked-prefill --enable-prefix-caching --skip-mm-profiling
    • --mm-encoder-tp-mode data
    • --reasoning-parser minimax_m3
    • --tool-call-parser minimax_m3 --enable-auto-tool-choice
    • --trust-remote-code
    • --tensor-parallel-size 4

    With these, vLLM reports the model as Detected ModelOpt NVFP4 checkpoint, uses B12X MiniMax M3 MSA attention, and the B12X NvFp4 MoE backend.

  4. NCCL/RoCE environment. On DGX Spark with a CX7 fabric, the values that mattered were:

    • NCCL_IB_DISABLE=0, NCCL_NET=ib, NCCL_NET_PLUGIN=none
    • NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 (the “twin” RoCE HCAs on the f0 port)
    • NCCL_IB_GID_INDEX=<your RoCEv2 GID index for the fabric subnet>; discover with show_gids rocep1s0f0 1 and pick the row whose TYPE is v2 and whose IP is on your fabric subnet. The exact index depends on your host’s GID table and must be re-discovered per host.
    • NCCL_IB_MERGE_NICS=1, NCCL_IB_QPS_PER_CONNECTION=8, NCCL_IB_SPLIT_DATA_ON_QPS=1, NCCL_IB_PCI_RELAXED_ORDERING=1
    • NCCL_NET_GDR_LEVEL=LOC, NCCL_CUMEM_ENABLE=0, NCCL_IGNORE_CPU_AFFINITY=1, NCCL_ASYNC_ERROR_HANDLING=1
    • NCCL_SOCKET_IFNAME/GLOO_SOCKET_IFNAME/UCX_NET_DEVICES set to the fabric Ethernet interface (e.g. enp1s0f0np0)
    • Fabric MTU 4200 end to end (host interface, switch port L2MTU, switch port MTU), with active_mtu 4096 reported by ibv_devinfo -d rocep1s0f0

  5. The --no-ray flag in step 2 is what makes the b12x runtime metadata check pass. If you skip it you get a startup error related to B12XPagedAttentionScratchCaps.__init__ missing copy_runtime_metadata.

Startup Checks

You should see all of the following in the head node logs:

Detected ModelOpt NVFP4 checkpoint
Using B12X MiniMax M3 MSA attention
Using 'B12X' NvFp4 MoE backend
FORCED_NCCL_VERSION 23007
vLLM is using nccl==2.30.7
NCCL version 2.30.7+cuda13.2
Application startup complete

If FORCED_NCCL_VERSION is not 23007, the LD_PRELOAD/symlink chain did not take effect and multi-node collectives will hang.

Notes

  • Cold startup is long… roughly 10 minutes for shard loading plus several more minutes for profiling, KV cache allocation, B12X MSA warmup, and CUDA graph capture (FULL mode is the slowest to capture). Multimodal warmup adds another 2 to 3 minutes. Total cold start on the head node was about 15 to 18 minutes on my setup.
  • Multimodal… send images as data URIs in image_url.url to avoid the model server fetching external URLs. Multimodal warmup completes successfully with --skip-mm-profiling on the chthonic build.
  • Performance snapshot from my llama-benchy run ( LLM - Spark Arena Benchmark ):
    • Generation tokens/sec, mean plus or minus std, concurrency 1: roughly 9 to 10 t/s across depths
    • Concurrency 2: 14 to 18 t/s for depths up to 32768 (higher depths skipped because KV budget exceeded)
    • Concurrency 5: roughly 26 t/s at depth 0
    • Prompt processing (pp2048) ranges from about 5000 t/s at depth 0 down to about 600 t/s at depth 262144

DGX Spark Cluster

image
image683×658 145 KB

2

Thanks for sharing this, appreciated to have the entire process documented rather than rely on other people’s docker images.

Do you have any thoughts on why performance is much worse than the spark arena version? Is it just because more layers have been preserved in bf16?

You could also try lukealonso/MiniMax-M3-MXFP8-NVFP4 which converts the preserved bf16 layers to FP8.

Waiting for my 4th spark to arrive, otherwise would give it a go myself.

3

that what I’m afraid of, minimax m2.7 was more than fine on TP=2 gb10, m3 is larger and even on TP=4 gb10 PP/TG numbers does not look awesome. Maybe some optimization will come, but I think we are stuck on 9-10TPS TG without drafter.

4

sparkarena/Minimax-M3-v0-NVFP4 is likely faster because it combines broader quantization coverage with a serving stack (sglang) that is better optimized for the DGX Spark NVFP4 workload. More BF16-preserved layers in lukealonso/MiniMax-M3-NVFP4 are a big contributor, but the benchmark results show the runtime and decode path are also major contributors.

5

I suspect more optimizations will come along not to mention the NVFP4 quant that will likely come from Nvidia. This model will be my daily driver moving forward and I will continue to find ways to improve throughput.

7

How’s quality so far for you ? In agentic it was worse than mimo v2.5 for me, currently optimize mimo but only for 2 Nodes

8

I can honestly say this is pretty good for my uses. I’m running the recipe from sparkarena, I have played around with a vllm recipe, but Ive not had enough time this past week to spend too much time with that.

I’m using it for DFIR, so having a bigger context is really useful, this is a good test for what I am building.

Initially I thought the reasoning was a problem, but now with some decent amount of time using the model I actually like that the fact the reasoning helps me understand why it came to that conclusion. I can push back in the instances it was wrong and get it to do change its opinion or to go back and look for more evidence.

The only coding I use it for currently, is to create smallish scripts when it needs to create something to look at the log files.

9

I have 4 gb10s and I use it in 2 pairs or 1 pair and 2 single mode. Minimax m2.5-2.7 was niche for that use case and delivered fine long assistant loops with some code researching/adjusting (yet main coder is qwen3.6-27b at the moment, served in TP=1 or TP=2 depending on plans for the day).

At quad node, I go for glm-4.7-nvfp4 max context, or glm-5, 5.1, 5.2 min context (or compressed KV cache, K Q8-> V Q4). MiniMax M3 is not good enough to get this quad node niche (at least vs GLM), you will get better results by combining minimax TP=2 + qwen TP=1/2 or minimax TP=2 + ds4f TP=2 (minimax workhorse, deepseek needle seeker/reviewer). That of course unless you use tasks from claude/codex fusion with local models.

10

Great question - I am updating my system prompts to be more inline with MiniMax-M3 prompting. So far, my anecdotal results have been positive. My primary use case involves VL capabilities, so MiniMax-M3 meets input text and image requirements unlike many of the other usual suspects that can fit in 512GB.

11

This is a good way to look at the reasoning transparency. I use it as an early indicator of instruction following.