DGX Spark 4-8 Cluster Switch Options post MikroTik CRS-804 \ MikroTik CRS-812 Market wipe out

Look like they have some in stock again I just checked and just ordered it let me this time selected 2day shipping an extra 70 bucks and its already marked for shipping this may come before my 4th node!

That’s great news! My 3rd and 4th Node are still sitting in customs since Sept 24th

Here’s to hoping someone realizes people need these to run operations :'(

Hope your 4th node arrives with if not shortly after!

Right now, Multilink have 1 in stock. If anyone trying to get one. MikroTik CRS804-4DDQ-hRM 400G Switch 4×QSFP56-DD, Dual 10G, RouterOS v7, 1U Rackmount

I saw the October 22 Technical Blog that captures up 1, 4 and 8 Sparks through Direct connect, Ring and Switch.

All I need now is for my Sparks to get released from their Sept 24 customs hold and come home…

Till then, this looks promising.

I wonder with these switches since the ports are 400GiB does that mean that you can connect up to 8 DGX on them?

You can connect up to

  • 16 Sparks on a single CRS804 using a 400G to four 100G split DAC cables
  • 8 Sparks on a single CRS804 using a 400G to four 200G split DAC cables
  • 4 Sparks on a single CRS804 using a 400G to four 200G DAC cables

What cables did you have good success with? Are you using a Y to put 2 sparks on each 400GiB port? Any speed issues doing that vs. going wiring each spark to its own port?

The second Spark bundle hasn’t arrived yet to qualify results for 4 Sparks on the CRS804. They’re still a month late from the planned delivery because of a customs hold.

These are the Cables I have. I bought different lengths for flexibility in placement only. You can go with the shorter ones.

I plan on using the Y cables for a dual-rail-network setup for 4 Sparks connected to 2 CRS804 switches.

CRS804 Switch 1 Port 1 400G ↔ 200G ↔ Spark 1 ↔ 200G ↔ 400G Port 1 CRS804 Switch 2
CRS804 Switch 1 Port 2 400G ↔ 200G ↔ Spark 2 ↔ 200G ↔ 400G Port 2 CRS804 Switch 2
CRS804 Switch 1 Port 3 400G ↔ 200G ↔ Spark 3 ↔ 200G ↔ 400G Port 3 CRS804 Switch 2
CRS804 Switch 1 Port 4 400G ↔ 200G ↔ Spark 4 ↔ 200G ↔ 400G Port 4 CRS804 Switch 2

What’s the benefit of connecting 4 sparks to 2 switches? … or did you mean 2 ports?

Fail-over, orchestration and experiment pushing dual-port dual-rail speed optimization.

I want to see if using 2 switches has any benefit beyond contingency fail-over.

I wonder if there is a way to team both 200GiB ports of a DGX to a single 400GiB port of the switch and provide 2x the bandwidth.

I hope so. I believe someone was able to unlock ~ 200G per port vs. the earlier ~100G per port speeds.

I’ll need to go back through the old thread. I recall bonding proposed but I don’t recall the results.

You can do 200G per port as I done that between two directly connected nodes.

You will however crash straight into the 256G bandwidth limitation of the 2x PCIe 5.0 x4 lanes going from the CX-7 to the GPU/CPU.

There is added complexity of having 4 subnets configured that will require all your recipes to have additional NCCL config for 4-way load balancing is another set of fun. You may also run into some NCCL race conditions if you are lucky as well.

As for failover, the LLM serving engines are designed to crash if the connectivity is disrupted. IE to fail over or fail from two to one switch means restarting the LLM.

I run dual 100G switches and as such could work around the need for 4 subnets:

Thanks for sharing @mashie and yes the set up will be both tricky and I expect friction.

Have you tried to override the engine design to fail-over through your second switch?

I don’t think there is a setting to tweak in this regard, the risk of corrupted data is too large.

In a large setup you have multiple clusters and a LLM router in front. This is where the intentional crashing makes sense as the LLM router will detect the LLM crash and send any new requests to the remaining LLMs still serving.

Testing the 804 and probably I am missing something. I have 4 DGXs, one plugged to each port, and they seem to be doing 109G as opposed to 200. What am I missing?

You can configure Y Cables to run 400G to two 200G per Spark

Even at all-to-all which stress tests the 4-node sparks it performs well

I have my TP4 configuration running perfectly, though the switch does get loud at times. It is bearable, maybe because I have my GX10 sitting on top of the switch, separated by a thin lap desk so it is not directly on the switch case, since I only have 1.5-foot cables which are extremely short. Got DSV4.1 1017 running multi concurrency like 10 sessions 1m tokens each at like 80ish tps using vllm. I’m glad I found that “like new 4th node” at only 4k looks like someone had buyers remorse it was mint.

I had the same issue with the very short cables and sparks on top of the switch. It sounded like an airplane taking off I lifted the sparks above the switch as much as I could, added some fans and it has been very silent after that.