I finally managed to start the full moonshotai/Kimi-K3 on 16x GB10 cluster, on a MikroTik Switch CRS804-4DDQ with 4x 400-to-4x100gbit breakout cables running with dspark runing on coherent corpus llama-benchy at 21t/s avg and 38t/s peak, 750tps prefill, as first try with dspark enabled using Inferact/Kimi-K3-DSpark. looking forward to improving this. I’ll be adding image and instructions to git soon: model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms) KIMI-K3 pp2048 @ d4000 654.89 ± 0.00 8288.20 ± 0.00 8286.85 ± 0.00 8288.20 ± 0.00 KIMI-K3 tg1500 @ d4000 21.71 ± 0.00 37.00 ± 0.00 KIMI-K3 pp2048 @ d16000 758.78 ± 0.00 21456.92 ± 0.00 21455.57 ± 0.00 21460.88 ± 0.00 KIMI-K3 tg1500 @ d16000 25.39 ± 0.00 38.00 ± 0.00 Tool-bench looking ok: Model: /root/models/models115/Kimi-K3 ││ Score: 91 / 100 ││ Rating: ★★★★★ Excellent

What’s new in v5 RoCEnante TP+DCP collectives: one-shot all-reduce (TP) and all-gather (DCP) over the RoCE v2 fabric, replacing NCCL for the small decode-size messages. Fused verify TILE8/TILE4 + nst6/nst3: 8/4-row fused spec-verify kernel, 6 or 3 speculative tokens, DCP-un-gated. Adaptive draft depth (opt-in recipe): acceptance-length-driven draft depth, floor 4 default (VLLM_DSPARK_DYNAMIC_DRAFT_DEPTH=1). >10 other fixes and optimization mods Prebuilt image: Package gb10-vllm/kimi-k3 · GitHub Guide to build the image and start Kimi K3 on v5 image: gb10-vllm/kimi-k3/v5 at main · ciprianveg/gb10-vllm main/kimi-k3/v5 vLLM inference solutions for NVIDIA GB10 (DGX Spark, sm121) — KIMI-K3 B12X_MLA + DSpark, GLM-5.2 - ciprianveg/gb10-vllm

Posting my results bringing up Kimi K3 on 16× DGX Spark (GB10), one GPU per host, native-mp (no Ray) — thanks @ciprianveg for the image + recipe. Full numbers at several settings first, then the gotchas that cost me the most reloads. Setup: kimi-k3:v4-prd, Kimi K3 mxfp4 (~1.5 TB), TP16, --gpu-memory-utilization 0.881, --kv-cache-memory-bytes 5000000000, marlin MoE, B12X_MLA, instanttensor, fp8 KV, --distributed-executor-backend mp. Results Transport was the biggest single factor. My container was silently running NCCL over TCP sockets until I added --privileged: config single-stream decode default (NCCL on TCP sockets) 1.3 tok/s + --privileged + NCCL_NET=IB (RDMA) 8.9 tok/s (~7×) No-spec depth sweep — DCP8 vs DCP16 (single-stream, tg=1500): context DCP8 DCP16 4K 8.75 8.75 50K 5.75 6.22 (+8%) 100K 5.12 5.33 (+4%) 250K 2.70 2.95 (+9%) concurrency ceiling 5.34× 10.0× Concurrency — DCP16, no-spec, max_num_seqs=8 (my production config): 16/16 concurrent requests OK, peak 13 in flight aggregate 36.76 tok/s (peak 48); 110 tok/s total incl. prefill mean TPOT 196 ms; ~7 GB free held (gmem 0.881, no OOM) Speculative decode is highly workload-dependent (single-stream): workload method decode draft acceptance coding, temp=0 dspark ~22–24 (peak 23.9) ~0.9 (accept len ~5/6) general/chat, temp=1 dspark ~5–9 (regresses) ~0.2 coding, temp=0 dflash 5.4 (regresses) low dspark is ~2.5× over no-spec only at temp=0; at temp=1 it’s break-even-to-worse. Learnings --privileged (or mount /dev/infiniband) — otherwise NCCL silently uses TCP sockets. Verify with NCCL_DEBUG=INFO → “Using network IB”. This alone was ~7×. --kv-cache-memory-bytes is the concurrency unlock. With no cap, K3’s Mamba/KDA hybrid cache balloons to fill the pool → stuck at max_num_seqs=1. A ~5 GB cap got me to seqs=8. Benchmark spec decode at your real temperature. If your client forces temp=1 (e.g. Claude Code with extended thinking), dspark gives no benefit — no-spec is faster and keeps concurrency. On separate single-GPU hosts, use the --disable-custom-all-reduce FLAG for dspark. The b12x PCIe-IPC spec buffer is same-host CUDA-IPC and crashes across nodes; the flag routes it over NCCL. The VLLM_ENABLE_PCIE_ALLREDUCE=0 env is a different code path and did not fix it for me. 16-rank scale gotchas: --ulimit nofile=1048576 (TP+DCP full-mesh gloo → “Too many open files”), and NCCL_ALGO=Tree breaks the InstantTensor loader’s ncclAllGather (“invalid usage”) — leave NCCL_ALGO unset. DCP16 on a single non-blocking switch: ~ties at low context, +4–9% toward 250K, and ~doubles the concurrency ceiling (KV sharded 1/16). Keep --gpu-memory-utilization ≤ 0.90. 0.985 dropped free memory to ~187 MiB and wedged nodes (sshd couldn’t fork → physical power-cycle; GB10 has no BMC). RoCE NCCL env I settled on: NCCL_NET=IB, NCCL_IB_GID_INDEX=3, NCCL_IB_DISABLE=0, NCCL_PROTO=LL,LL128,Simple, NCCL_BUFFSIZE=16777216, NCCL_MAX/MIN_NCHANNELS=2. MTP isn’t available — the checkpoint has num_nextn_predict_layers=0. Net: single-stream matches what others here report (same GPUs) — my wins were fixing the silent-sockets fallback and using DCP16 for concurrency (~37 tok/s aggregate).

If you need better temp 1 spec results change greedy to probabilistic

To answer question on practicality of the 16 node cluster, I drove this config for about a week and on multiple occasions, through sessions which blew through my context and compacted. Stability is very good, but I went back to an 8 node configuration with GLM-5.3-NVFP4 and 4 node on GLM-5.3-Flash. The other 4 node are serving Qwen3.8-Flash-Next for very complicated vision tasks in my domain (this is the best model for that). To be honest, this setup already solves a lot of my needs and Kimi K3 is evidently better, but not by much if I define the task very well (even I can get by with Qwen3.8 if I put in some human work and go into nitty-gritty). Goes to what @ciprianveg said… GLM will likely solve mostly everything, while smaller Flash-like models can nail 70-80%. I appreciate the flexibility in bringing Kimi K3 in when it is absolutely necessary, and it is nice to know that I will be ready when K3.1 (or whatever successor releases). I would say if you are looking to build a 16 node cluster just for this, really think what you want to accomplish first. But I will say that running 8,4,4 config or 4,4,4,4, or any number of combination for specific models does bring a lot of flexibility.

It’s like you are sitting next to me! I’m coming to the same realization…and the throughput is not nearly there with kimi k3 on a 16-node cluster. I also thought a data center switch would help but it really didn’t…the throughput is there but the internal pcie connection for the dual connectx-7 ports is the bottleneck as usual. A dual rail crs804 is more than enough. I am bringing up glm5.3 int4-int8-mix (4x4-node clusters) atm. However I’d like to try different configs too. Are you following specific spark-run recipes for the models? Do you by chance have links to them?

No spec v5 main model speed so far dcp 16: 4K 9.55 tps 50K 8.89 tps 100K 8.27 tps 200K 7.28 tps 300K 6.89 tps Speculative multiplies 2-3x depending on context type

V5 coming by tomorrow, +cca 20% vs v4: Coding: game-bench 1 2 4 8 (3000-token runs) C1: 29.81 tok/s C2: 42.00 agg / 21.00 mean stream C4: 58.00 agg / 14.50 mean stream C8: 87.12 agg / 10.89 mean stream Prose: llama-bench coherent corpus model test t/s peak t/s KIMI-K3 pp2048 @ d4000 880.35 ± 0.00 KIMI-K3 tg2048 @ d4000 23.59 ± 0.00 46.00 ± 0.00 KIMI-K3 pp2048 @ d100000 812.56 ± 0.00 KIMI-K3 tg2048 @ d100000 21.13 ± 0.00 33.00 ± 0.00 KIMI-K3 pp2048 @ d200000 848.73 ± 0.00 KIMI-K3 tg2048 @ d200000 20.03 ± 0.00 38.00 ± 0.00

That’s awesome! I’m looking forward to seeing what you did to get the boost. I’m still working on getting myself set up with 16x. I can’t wait to test it out once I’m there! Hopefully within two or three weeks.

Would the RoCEnante be useful for any other models/cluster sizes or is it only for when you reach 8+ nodes?

At 8 nodes Rocenante helped more than at 16. So i think 4 nodes will benefit from it.

Full Kimi K3 running on 16x GB10 cluster

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects

ciprianveg September 20, 2026, 5:24pm 143

What’s new in v5

RoCEnante TP+DCP collectives: one-shot all-reduce (TP) and all-gather
(DCP) over the RoCE v2 fabric, replacing NCCL for the small decode-size
messages.
Fused verify TILE8/TILE4 + nst6/nst3: 8/4-row fused spec-verify kernel, 6 or 3 speculative
tokens, DCP-un-gated.
Adaptive draft depth (opt-in recipe): acceptance-length-driven draft
depth, floor 4 default (VLLM_DSPARK_DYNAMIC_DRAFT_DEPTH=1).
>10 other fixes and optimization mods

Prebuilt image: Package gb10-vllm/kimi-k3 · GitHub

Guide to build the image and start Kimi K3 on v5 image:

Future of the DXG Spark

Topic		Replies	Views
Kimi 2.6 and Qwen 3.5-397B -FP8 on 8xGB10 cluster DGX Spark / GB10	28	2791	May 29, 2026
6x Spark setup DGX Spark / GB10	112	15220	April 25, 2026
Running Kimi K3 on 24 DGX Spark Systems — TP24 vs TP8/PP3 Advice DGX Spark / GB10 Projects ai , llm , agentic-ai , dgx	39	4691	August 14, 2026
4-node DGX Spark Cluster with DeepSeek V4 flash-0731-Dspark benchmark (prefill=2,500 t/s, decode=90 t/s) DGX Spark / GB10 deepseek	4	1736	August 22, 2026
Qwen3.5-397B-A17B-int4-AutoRound - 4 x db10 node - updated results 37 - 94 tok/s DGX Spark / GB10 clustering , spark	26	2708	April 28, 2026
Kimi K3 and the Ceiling of GB10 Clusters DGX Spark / GB10	70	11700	August 23, 2026
Should we as a community gofundme one Spark for Eugr's nightly builds? DGX Spark / GB10	51	2866	April 1, 2026
On the Uniqueness of the DGX Spark for Running SotA Models in Unconventional Environments DGX Spark / GB10	90	3236	August 30, 2026
Two multi-node DGX Spark wins: RoCE 2× inference throughput + Qwen3.5-397B-A17B-NVFP4 serving (with SM121 CUTLASS patch) DGX Spark / GB10 Projects	5	1924	July 21, 2026
Working recipe: MiniMax-M3 NVFP4 at TP=3 on 3x DGX Spark (no 4th node) + the OOM fixes DGX Spark / GB10	34	2930	August 8, 2026

Full Kimi K3 running on 16x GB10 cluster

Related topics