Based on the performance of the current open-source models, I don’t see any gain in large models with severely compacted accuracy to deploy on a 2 or 4-node DGX cluster. For example, gemma-4-31B-it with mtp would be a fair choice for a 2-node cluster. However, we cannot deploy another model on it simultaneously unless we lower the gpu_memory_utilization.
Even though Qwen3.5 397B A17B and many other models are better than gemma-4-31B-it[^1], we cannot deploy the base version without quantization accuracy loss.
I believe that encouraging ordinary individuals to purchase more than two DGX devices to deploy models is unsustainable because the cost remains unaffordable for most people. I find it hard to believe that when Sam Altman was an undergraduate, he already had access to over a dozen graphics cards. Today, with the advancements in AI technology, the average number of graphics cards owned by many university students in China is negligible. I believe this is also true for the people in the vast majority of countries.
What should we do to make this world a better place?
Or to put it another way, I’ve seen far too many different model deployment methods and test results. But these results all focus on throughput and speed, while shouldn’t the real focus be on output quality? Or problem-solving ability? Shouldn’t we add a ranking based on problem-solving ability?
Of course, a model’s problem-solving ability is clearly closely related to the abilities of the people using it. But from this perspective, it’s difficult to provide a better evaluation method.
Empirical data suggest that large models with aggressive quantization perform far better than smaller unquantized models with weights of same size in gb. Far better. But everyone has own recipe to cook the cat. No point to argue. Just FYI cloud api inference has models quantized mostly to q4 and lower, including sota frontier models from top3
There is another discussion on whether upgrading from 2x to 4x DGX Spark/GB10 units is worth it. To save your time, here’s a summary of what they’re talking about and their conclusions:
Main Topic
Is a 4x DGX Spark cluster worth the ~€10k investment over a 2x setup?
Key Points Discussed
What 4x Sparks Enable (vs 2x)
Minimax M3 (~500B+ model with 1M context window) — the main model that actually requires 4 units
Qwen 3.5 397B with comfortable context headroom (runs on 2x but with limited context)
GLM-4.7 / GLM 5.1 in NVFP4 format
Large FP8 models for quantization quality testing
Running multiple different LLMs simultaneously (e.g., cross-review between models)
What Remains Out of Reach
Near-frontier models (~1T+ parameters) — still impossible
Most 500B-750B models are “painfully slow” even on 4x due to diminishing returns
Sparse attention (required by newer large models) is not supported on GB10/SM12X
Performance Reality
Minimax M3 on 4x: ~19-20 tok/s decode at 500K context (usable but not fast)
NVFP4 recipes can hit ~24-27 tok/s on single, ~40 tok/s on 2x
Speed gains from 2→4 units are diminishing
Conclusions from Participants
User
Stance
Key Argument
0rand (OP)
Cautiously pro-4x
Math works out vs cloud rental (~29 months to break even at 1h/day); local = better tool eval scores, no prompt injection risks, sensitive code stays private
Teason2026
Skeptical / “don’t FOMO”
Most use cases fine with 1-2 sparks; better to combine local + cloud inference; quality gaps between 2/4/8 spark models are only 5-10%
Ria33
Pro-4x for future-proofing
Budget permitting, why not? Lifespan ~3 years, resale value likely holds; can split 2+2 for different model families
truxnor
Pro-4x, no buyer’s remorse
Needed context headroom; M3 is “just about acceptable” speed-wise; plans to buy more to run multiple LLMs for DFIR log analysis
Overall Consensus
There’s no strong consensus, but the practical conclusion is:
4x is worth it IF you specifically need Minimax M3’s capabilities (1M context, strong tool use), run sensitive workloads locally, or need to host multiple large models. It’s NOT worth it for pure FOMO — 2x handles most tasks, and cloud hybrid is often more cost-effective for occasional heavy lifting.
The main tension: RAM prices are rising fast, making future upgrades more expensive, but the actual performance gains from 2→4 are marginal for most models due to architectural limitations of the GB10 platform.
hello AI model, how’s your tensors are doing today? multiplication still going? PS I hated 4d vector algebra in uni, looking back I should have studied harder
I just saw glm 5.2 has pretty good result in yet benchmark mostly. I hope it does good job in real life. Probably? 4 units can do 4 bit quant + 256k ctx..? (not sure though)
And in Europe, you might find less than10k€ for 2 to 4 units jump. recently I ordered 3600€ x 2 for extra units + 1145€ for switch, 258€ for 2 cables = about 8.6k€. It might slightly cheaper since I live in where has one of the highest VAT% in EU.
If b12x works nicely, then I hope there would be way to extend 4 to 5 or 6 instead of 4 to only 8 route. Keeping door open for expand if there is need coming is great here with Mikrotik crs804 switch path. Ive experienced too limited expansion with strixhalo enough.