Qwen 3.8 Max is on my test list next week as it looks perfect match for 16x cluster at q4.
I also didn’t understand how Deepseek manages to keep prices so low?!
Good news, pp2 tp8 finally works, it unlocks 800k+ context. Bad news is that it does it at 15 tps, but better than dcp 8. But in my pov this is the best solution for now, this is the path I will continue to improve till an int4 faster and smaller quant. I will be pushing the image and mods soon in git and hope some more people join in testing and improving. With a model so big, every change and retest is 1h+
But Kimi K3 is already 4-bit, so what do you mean by “till an int4 faster and smaller quant”? Great work, and 15 tps sounds amazing for running a model like that at home at barely over 2,000 watts. Makes me want to make the jump to 16 Sparks!
My goodness! Thank you! I was waiting for this confirmation! Context length is more important in my scenario and I am sooo close to being able to do what you’re doing now. Thanks for trail blazing!
I’m assuming you’re talking about a 240V 10A circuit, but does the Spark have power adapters that support 240V? Seems unlikely but maybe in certain locations? For typical 120V it’d need two 15A circuits, which is still awesome!
Pure int4 would be cca 1.4T instead of 1.56T, and testing in the past int4 is faster than awq, nvfp4..
I am running it at home..
Yeah, that’s what I was saying, that it’s awesome since pretty much any house can dedicate two 15A breakers to the cluster. 8x nodes per circuit. I don’t get how people still seem to be sleeping on how much of an advantage the lower power requirements are compared to data center GPUs.
I have been saying this to anyone who keeps suggesting other systems like DGX Station or just building a bunch of desktop GPUs. The power requirements for what you get for intelligence using the DGX Sparks is sooo attractive. Can’t understand why not everyone can think this way :D
It does seem that very few people really understand the usefulness of the sparks, I would say long may it continue, but the price of ram in general is just making everything really expensive!
That is amazing. K3 is so agentic you can leave it working for many hours, speed is not the blocker in my opinion, context and quality are more important factors as they work mainly unsupervised
My conclusion with the big kimi k3 model on 16 sparks after real usage today is that is not usable at agentic coding, at least not yet. Even if context can be pushed close to 1M with pp 2, even if for structured input and coding it reaches 20tps at low context, the speed gets down fast. At 64k is 8-9tps. So it is good that you can run it and ask it some complex tasks, but currently it is just too slow to be used in agentic coding. Maybe a better mtp/dspark appears for it, or some extra vllm optimizations appear, I will keep an eye on it, while having more hopes for next ds pro, glm 5.3 and qwen 3.8 max.
The speed in the first post in coherent bench 21+ looks like they where caused by looping with dspark acceptance 0.8-0.9, looping generation are easy for spec model to guess. After solving the looping issue, in real opencode usage, it runs at cca 15tps at low context. If anyone finds some way to improve this speed that I missed, let mw know.
How much of that you feel is just due to unoptimized vllm for sm121? I am hopeful that it will get better – so I am hopeful to your “at least not yet” :)
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|
| KIMI-K3-FULL-B12X-TP16 | pp2048 @ d64000 | 760.97 ± 0.00 | 78241.25 ± 0.00 | 78237.86 ± 0.00 | 78256.71 ± 0.00 | |
| KIMI-K3-FULL-B12X-TP16 | tg1500 @ d64000 | 8.78 ± 0.00 | 22.00 ± 0.00 |
For improvements, i feel dspark has lower than normal acceptance, if the acceptance increases and if decreasing speed with context is solved to work as it is for glm, than would lead to 20+ tps at 200k context, this starts to make agentic coding usable. I’ll be spending some more time on it in the next week, fingers crossed..
I’m sure it can be improved. Truly, great work. You’re the only one I know of showing how to run Kimi K3 on the Sparks so far. Hopefully more people with 16x clusters will join you soon. I’m definitely on the fence right now, but I’m considering going from 8x to 16x if it sees a little more improvement or the future large models show success on 16x.
Thanks for your work on the GLM 5.2 quant. In your testing, did you find that the full GLM 5.2 FP8 wouldn’t be possible? Really great job on the repo and I hope to see, or help with getting, more models added to it!
Full 5.2 GLM fp8 works on 8x cluster but with less context. On 16 it should work wo any issues. I kept the 4bitint8 version because it worked really well allready. I think a good improvement in going 16x cluster is to upgrade also the switch. On 16x cluster,100gbit limitation starts to matter. For me, because the 16x cluster is formed by putting together my 8x cluster with my brother’s 8x cluster, to verify how the huge kimi k3 works, it doesn’t make sense yet to add a bigger and noisier 10k+ switch yet.
What I will try in september when a new switch arrives is to link together 3 mikrotik 804 switches with 400gbit cables and use the remaiming 8 ports to link all 16 of them at 200gbit. The comm speed should be between 133-200gbit on tp16 and 200gbit on tp 8 pp 2
You can still get away with just 2 CRS 804 using a full set of the same 400G to 4x100G breakout cables you already use. You just run them in parallel connecting 100G to each Spark port for a total of 200G.
You can even tweak the network config so you use one PCIe root complex for each 100G port.
But the main thing is that 3 daisychained switch will add a lot of latency. With two switches in parallel the latency stays the same but you still double the useable bandwidth.
Good idea, will roce v2 know how to use both network cards? Or they should be bridged as one and use the bridge as cluster interface?