I’m looking into setting up a local test environment.
I was able to borrow an HP Spark from HP, but I have to send it back next week.
Currently, I’m considering buying two and connecting them using a MikroTik CRS804 DDQ. That way, I would be able to scale beyond two Sparks. I’ve seen some guides explaining how to set that up.
However, NVIDIA also recommends directly connecting two Sparks to each other, which makes them appear as a single GPU.
So my question is: will they also show up as a single GPU when connected through the switch? And what happens if I go beyond two? I’ve seen some videos of people using eight Sparks with the model distributed evenly across them.
What would you recommend? I won’t be using them for training. I’ll mostly run tests for our RAG setup and run some custom LLMs before deploying them on the main server.
No. What you will have is two Sparks connected by a very fast network. The OS doesn’t see this as one big, unified pool of compute and RAM.
What it does enable you to do is take advantage of tensor parallelism for inference with VLLM. This will allow you to run models larger than will fit on a single spark and increases throughput.
Also, you don’t need a switch. A single cable connecting the two units gives you the full bandwidth of the Mellanox interface.
Any reason not to start with direct connection when using two units, and if you expand beyond two units then buy the switch?
On your questions: I’m not sure what you mean with “appear as a single GPU”, but the general idea when clustering is that GPU and Unified RAM will be “pooled” (but not from OS perspective), therefore the importance of latency and bandwidth between the nodes. Given that the Spark’s ConnectX-7 interfaces are 200Gbps and Unified RAM/GPU bandwidth is 273GBps, not taking into account the network latency (and a switch is gonna add its own as a new intermediate/hop), you cannot expect the performance to scale linearly.
If you only want to use two Sparks, you don’t need a switch. It won’t give you any benefits, but will add to the latency => make things slightly slower than when connected directly.
It will not appear as a single GPU, it will remain a networked cluster, but it will enable to run workflows that can use RDMA (like vLLM/SGLang) efficiently. For inference, it means you would be able to use tensor-parallel and increase your generation speeds compared to a single node (and run larger models).
rdma of connectx is bound to the pci part of the cpu (arm). cpu and gpu share one c2c mem controller. so unified memory saves copies from cpu-ram to gpu-ram, but also is beeing blocked when cpu uses ram or network uses rdma via cpu.
no pool of memory, just a way to talk faster to the net. 5μs rdma instead of 0.5ms socket
network latency is the price you will compensate your savings by load sharing with. you can share only 50% of compute time with other nodes. 22% is pure network time. having a switch between 30% ? and every node of the cluster will increase the load on the network, so latency grows even if its low compared with 200 GBits (have no measurements, but it must be linear).
so clustering will give you more performance when compute time is higher (more active parameters, higher context), it wont if you just want to scale smaller chats. and each new node in the cluster adds only a fraction of his performance.
We want to be able to scale beyond 2. We will start with 2. But we already have multiple models that we need to run that we expect to need to go to 4 sparks in a couple of months. I do not have the budget to buy 2 data center servers.
Nvidia told me that if you connect them directly it will act as a single gpu. From their email: “you can connect two Sparks together, and that will act like a single GPU for mid-size workloads.“
I know about the performance. Its more to be able to run larger models. Nvidia told me this “you can connect two Sparks together, and that will act like a single GPU for mid-size workloads.“
Just trying to figure out if 2 sparks connected directly work the same as 2 sparks through a switch.
depends on the perspective. if you just see the “other” nodes as an extension and call the box a “gpu”, it might be true. but honestly, this misleads much.
there are a lot of people working on making sparkies scale from the first token on tp=2++, but as you say: at the moment its rather for running larger models than scaling that much.
if it scales by tp=x, you’ve already reached the hardware limits before. no one likes models that run with 10 token/s and scale well with larger context… best achievements are torchrun and continous patches (UF17) that allowes some scaling combined with drafters from the first token. but this is not mainstream yet. and of course it depends. the more layers your model have, the more pingpong traffic is between the nodes and this costs token time.
currently running single sparc. the second one arrived. cable arrived. everyting ready. atm it is running LTX-2. It is so much fun. I am afraid to connect them. I will check the tutorial.
Yeah, “appear as a single GPU” in that sense seems kinda misleading. In the example of 2 units, you will have 2x 128GB Unified RAM, a total “pool” of 256GB that can serve models that can’t fit in a single unit.
But because it is not a “pool” from the OS perspective, the model will be distributed on these 2 units. Let’s simplify saying half the data on node A and half on node B. The software layer - which distributed those data between those nodes - will then use required data on node A and node B through the network connection between them.
With that in mind, having the node A and node B connected either directly or through a switch doesn’t change much things (expect the small latency added by the switch): the switch is functionally transparent, it is just node A getting data from node B (or node C, or node D, etc.).
So on your question “trying to figure out if 2 sparks connected directly work the same as 2 sparks through a switch” → yes, just nuanced with the small extra latency from the switch.