Don’t want to totally derail the discussion on Kimi but does the BF16 cache work on GLM FP8?
For the Kimi setup, do you have a way of seeing network usage if the connections are the bottleneck
You are already configuring that today if you use a single 200G connection, the bit where you need to configure two different subnets across two of the logical interfaces to get beyond 100G. It would be exactly the same function, just that the traffic is spread across two physical ports instead of spread across two PCIe root complexes.
Bf16 should certainly work.
This is exactly how I have eight nodes connected to 2x crs804 switches. I use 400gbps → 2x200gbps breakouts.
I ordered the cable, it should arrive in 10 days. Any specific configuration you needed to do?
Configure the second switch as a copy of the first switch.
On the Sparks, by default all you have to do is apply another 2 subnets for the second port, so each Spark will end up with 4 subnets:
rocep1s0f0 port 1 ==> enp1s0f0np0 - subnet 1
rocep1s0f1 port 1 ==> enp1s0f1np1- subnet 3
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 - subnet 2
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 - subnet 4
Now in theory you could reduce that by two so worth a test.
rocep1s0f0 port 1 ==> enp1s0f0np0 - subnet 1
rocep1s0f1 port 1 ==> enp1s0f1np1
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 - subnet 2
Worth to try the second option on two nodes to start with, then see if you can run two IB throughput tests in parallel across those links. If you get 100G + 100G it works, if not well, it was worth a shot. Option one will work regardless.
Thank you. I will try that.
@ciprianveg I just saw this Kimi K3 fine-tune on HF: abacusai/Smaug-Agentic · Hugging Face
It apparently maintains model performance while reducing token usage. Personally I’d rather see the vanilla Kimi K3 have its performance boosted. For some reason I prefer official models, but figured I’d mention the fine-tune.
I prefer the original. I am doing some profiling to see where it can be improved. I will be updating as soon as I get a faster solution.
Posting in an epic thread. Carry on!
Could be worth giving this a go, system requirements are a lot less. It’s a custom quant with measured low KLD:
Luke (developer of b12x) made it, so optimistic it will be decent quality. Unfortunately I’m only in the 4x camp so can’t test!
I couldn’t understand why someone spending on 16 Sparks will cheapen on the switch. For a 16-node cluster the switch is the most important device. A 24 x 200G QSFP56 switch can be bought under 10K.
The CSR804 is being promoted on this forum but that device has a puny Marvell switch chip, and the RouterOS is a underwhelming.
Two reasons:
1. The 16 cluster is built up mainly for testing what are the pros and cons of using 16x vs 8x and is built by putting together two separate 8x clusters, each with its own switch, and 2. The big 24x switch is too loud to keep it at home.
I will be pushing an improved version today, I changed also the dspark to newer version from Redhat that brings some speed gains.
That’s amazing, can’t wait to see what you’ve done!
It started to be usable for me in opencode, still has room to be improved. Considering KIMI K3.1 should be even smarter, it deserves some extra optimization time till then.
TP 16 500k context:
1)(llama-benchy, coherent corpus, tg=1500, ctx 120000):
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|
| KIMI-K3 | pp2048 | 610.91 ± 0.00 | 3098.73 ± 0.00 | 3097.03 ± 0.00 | 3098.73 ± 0.00 | |
| KIMI-K3 | tg1500 | 21.05 ± 0.00 | 37.00 ± 0.00 | |||
| KIMI-K3 | pp2048 @ d60000 | 684.17 ± 0.00 | 81825.38 ± 0.00 | 81823.69 ± 0.00 | 81827.58 ± 0.00 | |
| KIMI-K3 | tg1500 @ d60000 | 13.04 ± 0.00 | 29.00 ± 0.00 | |||
| KIMI-K3 | pp2048 @ d120000 | 649.73 ± 0.00 | 170239.21 ± 0.00 | 170237.52 ± 0.00 | 170244.05 ± 0.00 | |
| KIMI-K3 | tg1500 @ d120000 | 11.31 ± 0.00 | 20.00 ± 0.00 |
- coding, structured content is cca 20% faster
-max-num-batched-tokens was set 2048 for 500k context, 4096 allows cca 400k context and 15% faster prefill
image here: Package gb10-vllm/kimi-k3 · GitHub
tool bench:
Score: 91 / 100 │
│ Rating: ★★★★★ Excellent │
│ Engine: vLLM 0.11.2.dev280+infernal.de04f08.cu133.torch213 │
│ Max context: 403,200 tokens │
│ │
│ ✅ 60 passed ⚠️ 6 partial ❌ 3 failed │
│ Points: 126/138 │
│ │
│ Quality: 91/100 │
│ Responsiveness: 11/100 (median turn: 12.1s) │
│ Deployability: 67/100 (α=0.7) │
│ Weakest: J Code Patterns (67%) │
│ │
│ Completed in 1441.8s │ tool-eval-bench v2.0.0 │
That’s awesome! I’m assuming that’s still using a single CRS804? Looking forward to seeing what using two does.
Yes. I hope in a week or two to connect them at 200gbit.
coming soon another 7-10% speed gain solution
Just wondering if you have tried pushing the number of images per prompt beyond 1 , or if it isn’t a priority for you now?