Unlikely, however one thing I will do over the next few weeks is to see if my theory of a fully meshed 4 node cluster is physically possible with this hardware. Should have the required cable for that proof of concept at some stage in the next two weeks (3-5 wees lead time sucks).
SGLang, PP=3 (pipeline parallel) — not Ray, not TP. You’re right that vLLM won’t do TP=3 on M3: the head dims don’t divide by 3 (64 attn / 4 KV heads), so tensor-parallel across 3 GPUs isn’t possible out of the box. That’s why I’m on sglang’s pipeline-parallel for now.
NCCL runs over plain TCP between the 3 nodes — RoCE wouldn’t init for sglang’s PP coordinator on these CX7 NICs (the 2nd communicator crashes ~90s in), so I dropped back to TCP. That TCP pipeline is the ~10 t/s ceiling: PP means the nodes take TURNS on a single request instead of splitting it, so single-stream stays slow. Concurrency is where it shines (~90-100 t/s aggregate at 16 parallel). Running 375K context at MEMFRAC 0.90 (MSA KV can’t fp8-compress, so bf16 KV is the limit).
The loops + the slow single-stream are exactly why I’m about to take a swing at Luke Alonso’s TP=3 commit (fb63c9aa). His “virtual sharding” pads the heads (64→96, KV 4→6) so they divide by 3 cleanly — which finally makes real TP=3 possible on a 3-Spark box, no 4th node. Downloading his NVFP4 quant now. The open question is whether the NVFP4 kernels run on GB10/SM121 (his card targets SM120). If they do, it should beat the PP=3 latency. Will report numbers.
Please keep us informed. This is incredible opportunity to get the hardware that can be used before price becomes astronomical.
For tp=3 you need to run the b12x developer’s VLLM fork which integrates it. I’ve got some more cables coming so will try it as well next week. It’s one of the release branches in here: GitHub - local-inference-lab/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs · GitHub
Thanks! Heads up though, I checked all 43 branches on local-inference-lab and none actually carry the M3 model code yet (just the b12x kernels + TP head-padding separately). Luke’s commit fb63c9a in vllm-project consolidates it all — full M3 model + TP=3 virtual sharding + b12x sparse-attn in one place. Building from that now, will report numbers.
Any idea if SGlang works well enough with GGUF to use the unsloth GGUF’s of M3?
tried IQ4_XS - result = 10.7 tok/sec
That’s workable. Just imagine how much quicker it can be made if we can get eagle3 working and optimize lots of the llama cpp/sglang software for the spark. Damn I miss fable 5! 1 day and it patched a total of 12 bugs in musubi tuner and training speed up by 2.3x! Absurd. And it enabled partial Gradient checkpointing so you can manually specify how many blocks to checkpoint. Just imagine if we still had that model all the things that could get fixed. I continued the chat after it was nuked and opus 4.8 on high struggled on everything when fable just looked at things and was like “oh yea look at that the bugs just right there here let me fix this” and boom fixed.
suspense is killing me!
still working on it
I got TP=3 to work but it is running slower than PP=3 but I think I know why and I’m working on that now I’m very very close. But TP=3 is a thing I’ll have the recipe up I just need to get the stream speeds up passed PP=3
Results still working Working recipe: MiniMax-M3 NVFP4 at TP=3 on 3x DGX Spark (no 4th node) + the OOM fixes
got error when start on 4x DGX spark
Minimax M3_log.txt (21.2 KB)
Sorry been away for a bit. We can look at it, for sure. With 256GB of ram we should probably target ~200GB size to leave enough room for context.
Unfortunately I only have one spark, so this might be a little tricky lol
NVIDIA just released nvidia/MiniMax-M3-NVFP4 · Hugging Face.