I hope so. It’d be a shame if they drop the DGX Spark line. I’d love to see a DGX Spark II right on schedule.
It would make no sense at all for Nvidia to drop the Sparks. The main driver for the Sparks existance is still as a dev boxes to easier make things work on the big siblings (although not offering identical instruction sets didn’t help for that particular use case).
And as we know a lot of the community use the Sparks just as a great inferance machines which I can’t imagine was the idea but a sale is a sale.
Hopefully the next two versions will be closer to the bigger siblings.
I hope so and I’d assume so. It’s nice to see them doing the interview with Spark owners. Hopefully that shows commitment.
A bit of market research never hurts so it provides what people want and need.
The original intended use of the Spark has changed, it’s obvious in all of the comments above and the treads. While we love quanting our own models and training, a vast majority of the use has shifted to local inference. SPARK II will need to address this reality.
I am sure there are people fine tuning and quantising on DGX Spark quietly around the place for validation purposes and then proceeding to rent B300 time in the cloud when they want to scale up. I just don’t think they are an active part of the community forums.
For sure, but probably also not the main reason for the Sparks selling out.
Well, if people owning Sparks for home inference ends up with patches getting pushed to VLLM to optimize running certain models, that work still benefits the data center hardware. The software stack being build around Sparks has been pretty insane, as is the effort in pushing a few more tok/s or benchmark points of out any released model on the hardware.
The problem with the AMDs and the Apple’s is no High Speed ConnectX-7 equiv that I’ve seem I’d love to see something in the 15-20K range that gives me connectX-7 so I can expand if I want to
Would be nice to see a model comparison on that Mac vs dual sparks running a recent recipe. I feel like the time to first token is slower in those Mac results vs Spark benchmarks, and if speculative decoding works at all on the Sparks they seem like they pull ahead even on a single thread. And if spec decoding isn’t doing much, Sparks still win on c2 or more.
But every comparison post I see is “Mac has 4x the bandwidth, it’s way faster” without any real world results.
I shared results multiple times for qwen 3.8 flash next. I think m5 ultra 256gb will comfortably beats 2xsparks to death on up to 4 concurrents. M5 ultra vs 4xsparks - m5 will be only superior at low concurrency, sparks scale GPU and b7s, m5 ultra GPU and bus is same on 256 and 512.
I am assuming based on the fact that m5 ultra is two m5 max clustered at the system bus like m3 ultra was 2x m3 max. Early results show 5090 level performance that track with my assumptions. Mlx stack was recently massively improved on prefill and decode on long context.
sparks scale perf well because GPU is massively choked by the slow ram and bus. Clustering let it breathe. Metal is not constrained, GPU is at max from the beginning. So concurrency adds very modest perf, mostly splits t/s between sessions.
oh, of course it’s a flame-war topic, but I can’t pass by…
- A 200-gigabit network gives an advantage ONLY under “all else being equal”
if there’s a device on the market with 4x the memory size and 4x the memory bandwidth, then in 90% of cases you simply won’t need the network at all (most participants have no more than 4 sparks)
let’s consider an option. For sparks, 200 gigabits allow scaling the cluster with a performance increase of about 50% per doubling
1x 270GB/s
2x 405GB/s
4x 608GB/s
8x 911GB/s
16x 1366GB/s
hurray! a cluster of 16 sparks managed to catch up with one M5 Ultra node! an unimaginably powerful cluster!
“I want Kimi K3.” Sure, easy, you need 4 M5 Ultra nodes with 512GB each. Not 16, but 4. “But what about 200 gigabits?” Use not TP16, but PP4 mode. The Mac’s network card is enough to pump PP4. Here’s a cluster for Kimi K3 with the SAME decode speed as a cluster of 16 sparks…
at the same time, you can make 4 fast nodes out of a cluster of Macs (the speed of each will be no worse). And if you divide the sparks into 4 clusters of 4, each will be slower than the 16x.
200 gigabits won’t save you from competitors with larger and faster memory. Small slow nodes are ridiculous. No matter how many bicycles you connect, they will remain bicycles and won’t turn into a car.
I understand that Macs may not have as good a software stack today, but if, as Orand says, it is developing and improving, then these devices will be more useful than a mountain of slow sparks.
I also like the NVIDIA stack. But be objective already, Spark is very slow, and 200 gigabits speeds it up very moderately and disproportionately to the costs.
Problem with tp mode is we not only try to run bigger models but also improve speed as it is slow. Macs fast, so they typically use pp for clustering and have no problems, it’s much simpler than tp. You can pp systems with different ram and GPU too. You can pp m3 ultra and m5 ultra, you can pp macbook with Mac studio. It just works
Seems like the M5 ultra still underperforms a similarly priced DGX Spark cluster in both pp and tg despite the higher memory bandwidth. At least based on what I’ve seen from recipes posted around here and on spark-arena.
That if is doing a lot of work there at the end.
A worthy Spark 2: 1.5x-2x current compute, at least192GB( ideally 384GB) LPDDR6 RAM, ConnectX 8 and a price tag under $8K USD. Still, well segmented away from DGX Station/data center solutions yet will keep competition at bay. The question is only when they will release something like this.
The thing is, if the box has 384GB LPDDR6, it is large enough to not need clustering to run a lot of cool models in the coming years. Yes, Kimi, Max, Pros will be too big, but all Chinese Flash LLMs will fit nicely. With CXMT started producing LPDDR6, it might give a chance to some enterprising Chinese company like Huawei or Xiaomi to package that low cost RAM and some domestic AI chip like Ascend 950PR to make a compelling inference server for the prosumer/SMB market. And they don’t have worry about cannibalizing data center sales.
People often put no effert optimizing stacks, most of my tests on m5 max are twice faster than what people generally post on reddit because they could not be bothered to spend time to research and test. Same with sparks, but to a lesser degree because we have a strong community here where good stuff gets shared fast, which mlx doesn’t have. Subreddits are full of ignorance.
As an example: many use llamacpp based engines on Mac, this is a disgrace, unoptimized as hell. Even influences do that, ft. Alex Ziskind. It’s so cringy I unsubscribed, even after initially being a member du to his early coverage of spark clustering. His tests are woeful
Ignorance is widespread everywhere. I remember visiting KTM forums, so much complains about weird gearing super Duke has. All it takes - put a smaller front sproket and it turns into a excellent tool for city and back roads. 15m job, 50 bucks. Yet many won’t do s..t but keep complaining. Cant wait for Ai to take over and end this bs for good.
This is not Spark II