I use my Spark to test how I can execute critical company processes locally that I have not been able to implement with external AI so far.
I am not a programmer or AI researcher, just a normal user with a very good technical understanding.
I assume that this could be Nvidia’s main target group for Spark and RTX 6000.
On the topic:
I would like to know what the minimum hardware setup is for 2 to 6 nodes.
Which models run on it.
What these larger models offer over smaller models that run on 1 Spark node alone (most likely GPT OSS 120b).
What throughput can be expected, ideally input/output and total tokens to make it clearer.
It would be very helpful if you could simply describe your current successful setup, i.e., number of Spark, switches, cables, etc., and the simple software setup.
It would also be very helpful if you could briefly describe what you do with it, i.e., company data analysis, coding, etc.
This could help other beginners or interested parties plan a setup.
Currently, I am wondering, for example, whether I should buy an Nvidia RTX 6000 96GB or a second or third Spark.
1) Networking (2 Sparks)
If you’re only connecting two Sparks, a direct-attach copper (DAC) cable between the ConnectX-7 ports is enough.
2) Networking (more than 2 Sparks)
If you’re scaling beyond two, you’ll want a switch. Some of the MikroTik options are solid—for example the CRS804 line: https://mikrotik.com/product/crs804_ddq2
3) Models / what to run
Personally I’m a big fan of gpt-oss-120b. It pairs really well with the Spark, but there are always new models worth checking out (multimodal support, different “feel”, better tool calling, etc.). There are also some great YouTube channels where presenters benchmark and compare them.
4) Expected performance
Honestly: it depends. Throughput will vary a lot based on the model, quantization, inference stack, context window, and whether you’re measuring single-query latency vs concurrency.
As one data point: on my setup (two Sparks, DAC cable), I recently saw about a 15% TPS single-user increase after moving to gpt-oss-120b with TP=2 (two devices). That’s still new for me though, and I want more validation before I call it “final.”
5) More info
There are great resources on this forum—searching the history will usually turn up deeper answers and real-world configs for each of these questions.
Thanks! I was allready able to link 2 units using a 50cm cable everything worked well, but with GPT OSS 120B the troughput was slower compared to one machine with SGlang. So I wonder if it would be better with 2 Link cables?
Stock vLLM is not optimized for FP4 on Spark yet. However, Christopher here made his own fork that has similar optimizations and can achieve 60 t/s on a single Spark and >70 t/s on dual sparks.
As for other models, here is a table I compiled based on my own experience. It is slightly old now as it’s missing some new models, but will give you an idea of what performance to expect:
Okay, there’s one thing I’m not entirely clear about. There are two ports, each with 200 Gbps, for connecting two units. Does that mean with two cables I have 400 Gbps if I aggregate both ports? And does that increase (doulbe) the network throughput and also output in total tokens?
No, both ports are connected to the same two blocks of PCIe lanes, so you won’t get 400G if you connect both, each will be limited to 100G then. You may get just a very slight performance gain if you connect both compared to a single one, but it will still be around 200G in aggregate.
Today I landed in another country for a conference. In my backpack I have a DGX Spark, which I will be tinkering with from my hotel room. I could have left it on my desk running quietly, and accessed it remotely; but, coming in a SFF, having it with me is a possibility I do enjoy. Also, while I use it for prototyping code I may want to run at scale, it typically drains no more than 90W while doing inference. I couldn’t have done the same, I am sure, with desktop class GPUs.
You shouldn’t. Just buy RTX 6000 Pro then. 8x performance for 2x price - no brainer. Unless you want something very low power that can quietly sit in the corner of your desk.