I’m curious how the performance differs (for LLM inference) if you took a model that fits on a single Spark (like Qwe 3.8 27B) and spread it across two. I assumed it would get slower, because there’s now a “slow” network in the middle, however I notice all of the top scores on Spark-arena for 27B are dual-spark.
I asked Gemini and ChatGPT and both told me it would get slower, but then changed their mind when I said about these scores (ofc, now I don’t trust what they say 🙃).
On one hand, I can see that you’d effectively have “twice the memory bandwidth”, but since tokens are generated sequentially I’m not sure how you’d take advantage of that.
So, how does the performance actually change going from 1->2, and what factors affect it?
There are two major factor, first is the memory bandwidth. When you do tensor parallel the model split into two smaller models so the amount of reading from memory is half so it should be faster, because the memory bandwidth is the key limiting factor for inference of LLM.
The second factor is the coordinations between the two or 4 DGX sparks. This should be very fast and have very low latancy. In general this is not a big deal for the connectX 200g network cards. So it’s effect is very limited unless you go for large number of dgx like 8 or more. Also it could have stronger effect when you do training.
Thanks! I probed Gemini and ChatGPT some more, and had them explain Tensor Parallelism (compared to pipeline parallelism) and it makes much more sense now :-)
It scales until it doesn’t. The “slow” network isn’t that slow… for 2 nodes. It becomes a potential issue if you try to scale above 4 nodes. The law of diminishing returns. At 16 nodes, the nodes has a average internode bandwidth of 12.5 Gbps excluding contention (yes, switch network shouldn’t have collision, doesn’t means it doesn’t have contention), and that is a SLOW network . At 16 nodes, in my personal view, full TP cluster is no longer a viable option, a hybrid TP/PP setup might possibly be an option but layer on the complexity and cost, vs the ROI.
You need to figure out your main objective. If it’s just for bragging rights, by all means go ahead , no further considerations needed.
Is this for a single user or multiple users ?
What’s the load like ? Concurrent multi-users , multi-agents 24/7 ? Interactive workload vs batch workload ?
What models are you running or planning to run ?
Do you understand the concepts of TP, PP and EP ? And if you do, which one or ones are you planning to use ? If you don’t, then go read up about them.
What’s the purpose of going with your own cluster vs cloud ? If there is no reason , then $10,000 would be sufficient for a 2x 2 years subscription. You can subscribe to the max for 1 US based + 1 china based subscription for redundancy in the event that US models block any tasks that you want to do, e.g. security audit.
They are fools. I had this exact experience back in April when got 1st spark and was considering a second. They kept insisting TP works for multi-gpu only and I totally should use PP but my speed will drop comparing to 1 spark. And we talk about geniuses in a datacenter, LOL. Mind you - both were paid subscription models, not free 1bit retarded Gemini in google search.
I think all chat-based subs are running extremely quantized models, 2 bits at best. API or coding sub get you better quality. But many tests using OpenRouter of paid models consistently show worse results than locally run analogs, even 4 bit ones. They all serve super quantized.
yeah and practical results 4 node and more user posts are not encouraging speed-wise. 2-3 nodes make sense, there are good developments on virtual model heads to be able to split across 3 nodes, but performance gains are mute, only ram benefit. I would totally get a 3rd node and connect with mesh topology, if there is a strong reason. but router - hell no
Oh mostly I’m just curious, not likely to buy another. I saw the Mac Studio specs with 4x the memory bandwidth and it just got me wondering how things scale 🙂
I did not, but after the first responses here I learned about PP vs TP. I hadn’t seen EP though, so just reading about that - thanks!
Just to cheer everyone up - we see insane progress since beginning of this year. Qwen4 alone and improving stack will turn our 2-node setups into equivalent of 4-8 nodes by the end of the year - production Qwen 4 is planned by the end of Q3. Current Next is just a tech demonstrator. Yet if you charge for TOP OSS model - Kimi K4 10T parameters is coming, start stacking your sparks lol
Actually, the question is more complex than it seems. A lot depends on the model (dense or MoE) plus the size of the KV cache that needs to be transferred over the network with TP2/4 (for 27B it’s 17GB at 260k at 16-bit; for DeepSeek it’s significantly less, so partly that’s why scaling is better there).
The correct answer is “it depends on factors”.
But for example, in winter I couldn’t get the speed of Qwen3.5 122b on two sparks the same as on one (int4 + MTP 50 tps). But 27B gave a stable 1.6x speedup (1->2).
Yes 25% but on the same checkpoint! But this is not why you get 4 nodes over 2 - you run much bigger model with hefty active parameter junk. And you see your speeds drop.
Remember those post on the forum - they invest into 4 sparks plus a router, put GLM 5.2 on and happy seeing 20 t/s and 200k context. Jeez
@JW2026 man, I am happy nobody banning you in this thread yet, the other thread where cultists gathered was insane :D
I am trying to convince myself that three is the sweet spot, because the latancy is minimum and no need for switch. The problem of scaling between tp 2 and 4 is already resolved with available scripts.
The only thing that bothering me is that there are some models that don’t fit even at extreme quantization like Q2.
The thing is - this is exactly what is wrong with this scaling - big models don’t fit - you add sparks, but now they are so heavy and performance scaling slope goes to diminishing returns - you will be able to run a big model but at maddeningly slow speeds
yeah, so you only buy RAM and in some cases it makes sense - for cache for example. Ideally we would want VLLM to serve cache only - there are implementation for DC systems, like LM-CACHE
So is 2 sparks kind of the overall sweet spot? Honestly I’m not sure if 2 is even necessary considering 1 spark models just seem to keep getting better and better, but I do want to try at least the next step up so I did buy another Spark. cable will be here Mondayish.
Yeah but DS4F is very fast on two already with DSpark - both prefill and decode. Likes of M3 would be interesting in theory (now with GLM its irrelevant) if they could fully utilize 3 nodes, or the GLM 4.3 Flash itself - especially prefill.
