Hi everyone, I currently have 24 NVIDIA DGX Spark systems and am planning to test the new Kimi K3 model across the full cluster. I would appreciate advice from anyone working with large MoE models, multi-node tensor parallelism, or DGX Spark networking. Kimi K3 is a 2.8-trillion-parameter MoE mode…

@ciprianveg what he means by “dual rail” is two independent RoCE fabrics, since each Spark has two CX7 ports.

For others looking at serving Kimi K3, I built this calculator: Kimi K3 Serving Calculator

[Screen Shot 2026-07-31 at 13.26.02 PM] Updated calculator with some 24 Spark analysis.

Exciting! I don’t think i would buy 24 sparks over a station… but.. I’m still waffling on getting my second spark anyways. Poor is me.

[image] Can’t wait for the review on this cute cluster!

Your fleet is hot. No seriously, how much heat does that stack generate?

It started by connecting two physical ports to the switch. But does to H/W limitations it produced same result as 1 physical port two logical splits. we did that to achieve ~190-220 Gb/s fast fabric speed. Thats what i mean by dual rail

We are using nvcr.io/nvidia/vllm:26.07-py3 (vLLM 0.24.0+092c4842.dev) with NCCL 2.30.7. All switch connections have been verified, and both CX-7 fabric links are confirmed operational at 200 Gb/s with consistent MTU settings. I dont think this image is sufficient to run kimi K3. But i also think t…

Lol pretty easy to accumulate $300k of hardware when much of it is review samples from manufacturers. STH has good coverage but they aren’t a reproducible example for people to follow. Following this developments in this thread closely though, OP, particularly the networking topology. Even if I ca…

Any news on your experiment. My Cluster installation got delayed in delivery :/.

Running Kimi K3 on 24 DGX Spark Systems — TP24 vs TP8/PP3 Advice

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects

aisling42 August 14, 2026, 5:43pm 41

@Summer_DGX_24 are you still working towards this? @ciprianveg just posted an updated recipe for 16x: Full Kimi K3 running on 16x GB10 cluster - #57 by ciprianveg

Topic		Replies	Views
Full Kimi K3 running on 16x GB10 cluster DGX Spark / GB10 Projects llama	82	6604	August 24, 2026
Nvidia releases Kimi K2.5 NVFP4! (1T, 591GB) DGX Spark / GB10	16	4421	May 7, 2026
Kimi K2.5 viable on a DGX cluster? DGX Spark / GB10	11	1806	March 11, 2026
Kimi 2.6 and Qwen 3.5-397B -FP8 on 8xGB10 cluster DGX Spark / GB10	28	2286	May 29, 2026
Working recipe: MiniMax-M3 NVFP4 at TP=3 on 3x DGX Spark (no 4th node) + the OOM fixes DGX Spark / GB10	34	2145	August 8, 2026
6x Spark setup DGX Spark / GB10	112	12285	April 25, 2026
Should we as a community gofundme one Spark for Eugr's nightly builds? DGX Spark / GB10	51	1970	April 1, 2026
DGX Spark performance DGX Spark / GB10	49	7237	February 13, 2026
DGX Spark + Qwen3-Next-80B: Proven Performance, But Missing Clear Path to NIM, TensorRT-LLM & Web UIs DGX Spark / GB10 cuda , nim , llama	16	5395	March 6, 2026
Three node Spark clusters (without a switch) are now supported in spark-vllm-docker and sparkrun! DGX Spark / GB10 llama	15	2952	July 19, 2026

Running Kimi K3 on 24 DGX Spark Systems — TP24 vs TP8/PP3 Advice

Related topics