Performance characteristics going from 1->2 sparks

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
22

Curious is as good a reason as any, you don’t need anything more than 2 nodes to test most of the technologies. 4 nodes if somebody else is paying for it or maybe you earned tons of monies from selling Micron shares.

I have not follow on the new Mac Studio but I’m guessing still 10G for networking unless you go with Thunderbolt, which probably limits you to 2 nodes or else PP. Assuming they restarts the 512GB line, I would guess they would price it around $15-20K , so $30K for total of 1TB of unified ram. Faster ram, probably less powerful GPU/NPU, xPU , good for inferences , not as much for training, not so sure about AI video.

The advantage of Mac Studio is that Apple is not motivated to tune down their performance to avoid competiting with their own products since the studio is the top end. They are not using faster ram (They were negotiating with CMXT for better prices and they were brushed off) , they just have wider lanes. Nothing to stop AMD or Nvidia to do the same IF they want to.

23

I has 5 thunderbolts so you can mesh 4 nodes at 80 GB/s. The hardware is good, but MLX is ■■■■. Large context slows down a lot. of course with clustered M5 Ultras you start with a high base, so the end results might be very good

24

But standard TP won’t work with 3 nodes, so more memory, probably slower single user/session performance. So it’s 2 or 4 if you want to use TP. Some interesting “hacks” / “developments” being done in this area but TP=3 is still niche . Not sure if it will become main stream.

25

It seems to me like the Spark is just still the way to go if you actually want to seriously use some decent models at home without setting up some huge 5 GPU hotbox (which isn’t even any cheaper), and especially if you need the VRAM that two of them provide. The Mac has its advantages but this ecosystem is just so active, I guess in part because it’s what all of the top players in the enterprise industry are using too, kind of. I’m glad I ended up choosing this Spark over the AMD 395’s.

Yeah just hurts to pay that much, but oh well. Hopefully it will be worth it.

26

Indeed, thats exactly the reason I still have 2 sparks, not 3 :D

27

I agree with 2 nodes being optimum but disagree with the statement that you paid $1500 for it. You do get a 200Gbps Xconnect but the Spark is now about the same price as a branded AMD AI Mini PC with 128GB ram. It’s probably just a couple of chips that cost Nvidia $20/= . Bottomline, get what fits your needs, not what some strangers on a forum recommend based on a general idea of what you wants. Learn about the pro and cons , engineering, design constraints , see if it applys to you individually and make a decision from there. Not " a couple of guys has post a working 16 nodes cluster, so that’s what I need …".

28

I have 4 nodes myself and it gives me more options with models now as well as going forward.

I also have a setup with just 2 nodes at work and that is working fine as well running DSv4 driving Hermes.

29

I have been summoned!? Ha!

It is correct that going from 2 to 3 or 2 to 4 gb10 devices is not a 1/3x or 2x increase in performance. Very little, if any computing scales like that. It also heavily depends on the architecture of the model being run. A qwen3.8-27b dense model doesn’t run the same across 2 or 4 as GLM 5.3 Flash or Qwen Flash Next. And even quant to quant, that dense 27b in my thread is slow, about 1.5x as slow or worse than newer quants, but its KLD is EXCELLENT and I trust the guy quanting it. So many variables.

All number of sparks work, for the most part, at this point. You simply use “padding” so that everything that is usually divisible by 2 is… well still divisible, you pad it up. The compute of the additional spark/s always increases performance, up to a point that I have not reached yet, nor seen on the forum or elsewhere. It physics and at this point in our inferencing journey, the 200Gb links are NOT the bottleneck, not even close. Is the increase “worth” it. Not for me to decide.

Finally, now that I’ve stream of conscious’d a response…

More sparks lets bigger models fit or the models that do fit, have more KVC room, room for speculator models (dflash, for example), more room for unified ram workspaces for things like DCP2+, etc. And again, they WILL run faster. See my thread. How much so is so variable…

Also keep in mind, my numbers are on a custom ‘ring’ overlay, without a switch. An a-typical setup. I do encourage people to try them. They perform and “feel” so much better in real usage, those benches are low! My preferred setup currently is:

4x GLM5.3flash in a ring with sparkcache
+
2x Qwen3.8 flash next no sparkcache (yet)

EDIT: I haven’t updated my threads, or maybe even the repo.. but 6-spark rings do work as well.

30

Dont forget you can also throw a reverse proxy in front of multiple sparks as well. If your goal is raw throughput, you might beat out coordinating the split of a model across spark’s links. Still constrained on total pooled memory and memory bus speed but you might be able to get more concurrency.

31

@Orand

Fixating on a single architecture like DSV4F as the ultimate solution for every use case (poor you) shows a fundamental misunderstanding of real world engineering. There is no “one size fits all” model. Machine learning workloads are defined by specific trade offs: memory bandwidth constraints, latency requirements, hardware topologies, deployment environments, and specific task optimizations. Pushing one model as a universal dogma on every single thread doesn’t add any technical value, it just derail valid technical discussions.

Spamming multiple threads with the same rigid ideology while attacking and insulting other developers isn’t constructive criticism; it’s just noise.

It is precisely this toxic behavior that has driven many talented researchers and developers away from sharing their breakthroughs, benchmarks, and custom implementations on this forum. When constructive engineering discussions are constantly derailed by fanatical preaching and hostility, the platform loses its value for the entire community.

If you have actual hardware benchmarks, profile logs, or technical analysis comparing scaling efficiency on different topologies, share them. Otherwise, hijacking technical threads to push a single-model agenda helps no one.

32

True, yet in most computing, you are dealing with overheads of clustering, not hard bottlenecks constraint by hardware.

The sweet spot for “clustering” is two nodes and four nodes (+switch) but diminishing returns from a raw performance perspective above 4 nodes. Obviously this depends on the specific models and configurations of clusters, use cases.

33

How is this relevant to a thread about going from a single Spark to dual Sparks? “Bragging rights” with dual Sparks? Maybe you got confused and posted in the wrong thread. There are other threads which are about 16 node clusters. You might want to check it out because a very similar account has been making the same argument as you :)

34

Sweet spot as in you get to run bigger models without losing too much in terms of performance per unit. At two nodes, the 200 Gbps (which is kinda slow by AI cluster requirements) is not yet the bottleneck it will be at larger cluster size. True enterprise clusters have multiple 800 Gbps or even 1.6 Tbps , and direct GPU to GPU networking, not a shared 200 Gbps link. Single user or small workgroup, single spark models works great.

35

Yeah I thought your threads are a good example for scaling, because you do provide the numbers for multiple spark configurations

When running such cluster, how bad is the CPU temperature on the head node? Does it get worse or is it the same load?

Yeah it seems MoE models scale better

36

I contend no switch is necessary at 4 nodes or even 6, but I am biased! haha

EDIT: oops haha, I had started responding to someone else mentioning a switch for 4x+

Most certainly, it really just comes down to that parameter count that needs to run through that slow ram (for the most part). So, these small, active parm MoE’s are excellent.

Really interesting question about the head node temp. I’ve yet to throw consistent monitoring on these guys (I know.. I know.. ) so I have no numbers to report (besides eyeballing during benchmarking to be temps are generally OK) So, to actually answer, I do not know. I expect it does run a bit hotter just based on it having to do a bit more work on scheduling, api handling, etc. I will find out!

EDIT2: Qwen3.8 flash Next is another excellent 2x and 4x spark model people should try. Even if that means using sglang instead of vllm, or vice versa. I’ve found it runs well on sglang on 2x sparks. Random experience for anyone reading at home.

37

Both prefill and decode meaningfully improve at TP=2 in a proper setup, but not by a factor of two. You get twice the effective bandwidth for decode and twice the available compute for prefill. You end up giving a chunk of it back because allreduce operations occur over the ConnectX link. But that allreduce is per layer, not per token.

My current Qwen 3.8 27b setup in NVFP4 does prefill and decode almost exactly 50% faster with TP=2 vs a single spark. Whether you consider that diminishing returns or not for a second Spark depends on what your objectives are. My use case really prioritizes optimizing prefill. I’m considering moving the workload to an RTX Pro 5000 72gb. Anyone want a deal on two sparks and a network cable?

38

Probably the best answer here, IMO. With a properly configured 2, 4, 6, 8 etc spark cluster… the 200 Gb/s links are generally fast enough that the gains in aggregate compute and memory bandwidth outweigh the collective/communication overhead. Typically, enough to end up in the “this could materially improve my use case” area.

Shipping to Texas? :D

39

Yes. I’m in MO.

40

I shared winning recipes people actually use for Qwen 3.8 Flash Next FP8 x2, GLM 5.3 Flash NVFP4 and Qwen 3.8 27B FP8 x2 in last 2 days. How did you benefit this community, friend?

As for DS4F - I currently have 2 Qwens, DS4F and GLM 5.3 Flash on my sparks plus Ling Flash, Qwens, and more models on Mac. So no, DS4F is not the only hope, Obi Van. They are more jedis around. DS4F is just the best gunslinger out of them.