I currently have 24 NVIDIA DGX Spark systems and am planning to test the new Kimi K3 model across the full cluster. I would appreciate advice from anyone working with large MoE models, multi-node tensor parallelism, or DGX Spark networking.
Kimi K3 is a 2.8-trillion-parameter MoE model with 96 KDA attention heads, 93 transformer layers, 896 routed experts, and an MoE intermediate size of 3,072.
Based on the model configuration and the current vLLM Kimi K3 implementation, the following two layouts appear worth testing:
Experiment 1: TP24 / PP1
96 KDA heads ÷ 24 ranks = 4 local heads per DGX Spark.
The KDA projection size is 12,288, giving 512 projection dimensions per rank.
Q, K, V, gate, convolution, output projection, A_log, and dt_bias all appear to shard cleanly.
The 128-dimensional f_a projection is intentionally replicated rather than tensor-sharded.
The generic Triton KDA path appears capable of processing four local heads.
However, four local heads do not use the currently optimized fused KDA decode kernel, which supports 12, 24, 48, or 96 local heads.
A larger concern is that 3,072 ÷ 24 gives only 128 MoE intermediate columns per rank. Current vLLM code appears to pad this to a minimum of 256, potentially creating substantial additional expert-weight memory usage.
TP24 would also require tensor-parallel collectives across all 24 machines for every layer.
Experiment 2: TP8 / PP3
96 KDA heads ÷ 8 = 12 local heads, matching the optimized fused KDA decode path.
3,072 ÷ 8 = 384 MoE columns per rank, so no minimum-width padding should be required.
Tensor-parallel collectives would be limited to groups of eight machines.
The possible downside is pipeline latency and bubbles, especially with low batch sizes or single-request inference.
I may also test TP12/PP2 because 3,072 ÷ 12 = 256 exactly, although the 93 layers would need an uneven 46/47 split and eight local KDA heads would still miss the fused decode kernel.
My main questions are:
Has anyone tested very large tensor-parallel groups across DGX Spark systems?
Is TP24 practical over the DGX Spark ConnectX-7 network, or will all-reduce latency dominate?
Is the MoE intermediate-width padding at TP24 likely to affect the actual stored MXFP4 weights, or only runtime kernel allocation?
Which NCCL, UCX, GPUDirect RDMA, or network-topology settings would you recommend?
Are there known limitations in the Kimi K3 KDA kernels when running with four local heads?
Would TP8/PP3 be the most realistic starting point for this 24-machine cluster?
Are there any current vLLM branches, containers, or patches specifically recommended for Kimi K3 on GB10/DGX Spark?
I plan to compare model-loading success, unified-memory usage, time to first token, decode throughput, prefill throughput, NCCL communication time, pipeline utilization, and whether any memory paging occurs.
Any advice, warnings, configuration examples, or benchmark suggestions would be greatly appreciated. I will share the results from both TP24 and TP8/PP3 once the experiments are complete. Lets get this thing working shall we !!!
tp 8 cant fly . its 1.5tb weights and they are 4bit already expect for the activations.. you need tp16 .. also tp 24 does not exist .. it scales with power of 2 so 1 2 4 8 16 32
idk if you are trolling .. but if you had that many devices this would be already acquired knowledge
Little bro TP8 PP3 not TP8 PP1. TP8 alone wont work of course. . Also, neither vLLM nor NCCL universally requires power-of-two tensor-parallel groups. TP24 is permitted when the model’s sharded dimensions divide correctly.
@Summer_DGX_24 what switch are you planning to use? CPUDirect RDMA is not available on Sparks so you’ll have to rely on RoCE to work with that, close to, 3 TB of aggregated memory. A CX7 200G RoCE fabric, single layer2 network will work fine. Didn’t try it, though!
As you add a GB10 to a cluster, your ROI diminishes, beyond 8 you should really be losing performance if you operate as a cluster. OK to have 24 and operate as 3 load balanced clusters of 8. But 24 in a cluster? No, if only as a proof of concept that you can run multi-tb models.
yeah, something to watch out for like @giles8 says, at some point there’s going to be a lot of traffic. I am assuming pp=3,tp=8 is probably going to win.
Hello Elasaco, thank you so much for your reply. I already have a build with 16 sparks connected with an RoCE RDMA enabled switch with 400 Gb/s and it supports dual rail too. Problem is each spark machine max speed is 200 Gb/s while enabling dual rail. I am right now working on adding the 8 sparks. The switch can take up to 32 sparks. I already played with nvidia guides and managed to run big models like glm 5.2. The problem with nvidia guides is that they rely on the docker image generated for vllm from here: vLLM Release 26.07 - NVIDIA Docs
and we do not even have stable offical vllm version that can support it, think of time needed to have nvidia adapt that :(.
I am a big fan man. In the past i used the build you made to fix sm121 with sparks and run the quantized glm model. I even added dual rail and extended it to 16 machines and used the fp8 version of the model and it still worked :). Right now i am working on Kimi K3 and getting the new 8 machines installed. You are awesome !!!
I will be giving k3 a try on a cluster of 16 this weekend, too. Not too optimistic about speed, but lets see.. I am hoping 15-20 tps with dspark but afraid of 5-10 :))
Thanks for the detailed analysis. Based on current DGX Spark clustering guidance and the present state of Kimi-K3 runtime support, TP8/PP3 appears to be the most practical starting point.
We recommend first validating the exact vLLM image/build on a smaller topology, then scaling up after confirming model loading, memory behavior, and NCCL communication are stable. For the network baseline, please use the official DGX Spark clustering guidance and verify MTU consistency across the CX7 fabric.
TP24 may still be useful as an experimental configuration, but it is probably better treated as a follow-on test after establishing a clean baseline.
If you can share your DGX OS version, vLLM image/tag, NCCL version, and switch model/config once testing begins, that will help narrow the recommendations further. Please keep us posted on the results.