Questions on 2-node and later 4-node dgx spark cluster

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi! Since yesterday I got my 2-node spark cluster running. Thanks for all the tips and pointers I found here in the forum :-) I documented my steps here:

I have a couple of questions:

Right now I have a single qsfp56 dac cable between both nodes on the first port. Only having assigned a single IP address to the enp1s0f0np0. The other interface on the first port enP2p1s0f0np0 is up as well, both have mtu 9216. Is this the optimal configuration ? So no ip on the enP2p1s0f0np0 ? I understood that nccl is using RoCEv2 ? where RoCEv2 is IP protocol, not Layer2 ? How can I confirm that it’s using the full 200gbps and not “only” 100gbps ? (see above link to section in SYSTEM.md doc)

I’m getting up to about 18tps on the MiniMax-M2.7-NVFP4P4 model, is that to be expected ?

Is it true that there is some extra speed to be achieved when connecting two QSFP56 cables ? I understand that both ports then both fall back to 100gbps, but there is still some gain ? In that case which interface gets an IP ? Only enp1s0f0np0 and enp1s0f1np1 ? or also the P2 variants ? and do they need to be in different subnets ?

I also purchased the mikrotik CRS804 because i want to upgrade to 4-node. So if the dual dac is optimal per dgx, would that mean to use 4x qsfp56-dd cables, each going to two ports on each dgx ? Can i use the same 9216 mtu on the switch ?

thanks!

2

Using a second cable will only add complexity and no speed.

Physically it is 2 x 200G ports but due to the design they share 200G worth of total bandwidth across the PCIe links.

Also running a 2-node cluster via a switch may reduce the performance vs a single direct cable as the switch adds latency.

3

thanks @mashie

i’ve also found that i wasn’t using nccl.. now gave ip to both enp1s0f0np0 and enP2p1s0f0np0 and added envs for nccl, speed has gone x2 now :)

4

You can also use Sparkrun made by @dbsci to help with making sure your ports are configured correctly.