I had to, just need more memory to run concurrent models. What do you suggest, vllm with a gpu memory limit, or llama.cpp and just run models on each, without necessarily clustering, or a combination of both?
Welcome to the club ;). 2 sparks is better value than 1, as with a single spark you are not making use of that expensive 200GB/s interface, which will not only allow you to run bigger models, but also allow you to run smaller models faster.
Dont bother with llama.cpp, vllm is much faster, especially for agentic workloads, and containers / recipes from eugrâs repo make running vllm very straightforward to set up.
Yes, Ive been using his docker so far up to now, not experimented with concurrent models though, Im looking to run 1 large model, for deep reasoning, 1 multi modal model for image / computer use testing, and another model for straight coding. This will help me put it in a place (going from 1 to 2). Just thinking how to manage the memory concurrentlyâŠ
Welcome to the club. Also started with 1 around three months ago and now there are 8 running next to me.
Both scenarios make sense, personally my go to model is right now OpenAI GPT-OSS 120B (2-node) in a cluster setup. There is a cable **â**ThinkStation PGX QSFP Link Cableâ which is really cheap for connecting 2 sparks together. Using vllm with the help of spark-vllm-docker
I just told âI bought a mini-pcâ⊠No more explications needed, mini-pcâs are fun and cheap ;-)
Nvidia needs to add another playbook: What to do when wife ask you about DGX Spark:
1 unit: xxxxx
2 units: They are just mini-PCs (which is true!)
4 units: xxxx
8 units: The company bought them so they could make me work on weekends (âremember to burn the credit card statement, or erase the fileâ)
1 Unit: I need it to win contracts
2 Units: I need one for me and one for my clientâs work
3 Units: I need one for me, one for my clientâs work and one more to win more contracts
âŠ
N Units: I need one for me, N-2 for my clientâs work and one more to win more contracts.
How many clients do you have?
Tell the wife they are just a new version of Raspberry pi, my wife did not even notice the new little box
Same, just joined 2 family :D and thinking about third⊠Any thought for you for 4âth? or to much hassle with router etc :D
why not using 1 model (like qwen3.5-397b), it can be running concurrently with multiples requests?
Yes possibly but smaller agents may work faster.
Judging on responses by stock qwen3.5-122b - no I need specific short responses according to their expertise, not long diatribes about what the question may be
for 2x spark cluster minimax m2.7 awq is the way to go.
It is very good and works well with opencode. the qwen 3.5 122b fp8 recipe is also good but after testing both of them on a couple of the same prompts I found that minimax was superior.
Expect around ~40 t/s max, going down from there depending on context size. overall very usable
Yes, certainly when the second node arrives (still order âprocessingâ). I will first experiment will all the large cluster level models. My exposure with the single node has been just with one model loaded at a time and none of the cluster-only models have been in that testing. I have, however, found that individually medium size models are OK on coding and multi-modality inputs, and the larger models are also OK with more complex coding and orchestration (although a little slower). I will just need to find a model, or combination that fits my needs.