3 cluster Nvidia DGX Spark Cluster NCCL Not Working

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hello, I have 3 Nvidia DGX Spark servers. Originally, I had 2 Nvidia DGX Spark servers in a 2x cluster with an ASUS QSFP cable running nvidia/MiniMax-M2.7-NVFP4 and it was fantastic so I bought a 3rd DGX Spark to get more KV cache.

Now, I have connected the 3rd one. All three have sudo no password, ssh key no password, ssh hostkey checking disabled, firewall disabled, all fully up to date, all running the same firmware/CUDA/etc.

NCCL will not come up. I’ve been trying to use sparkrun which worked like a charm for 2x cluster but after debugging everything under the sun I can’t get the 3 nodes to boot any model I’ve tried. They call fail at nccl.

All links say they are up but odd networking issues persist. Like SSH not working between all IP’s in the cluster and at first the ports that the no-name Nvidia cable are in were always coming up with 1500 MTU. I manually changed this and manually configured the netplan to set 9000 MTU. However, didn’t help. I followed all instructions e.g. port0->port1 in a 3 nodes ring, wiped the /etc/netplan yada yada, and all that. Wiped all 3 servers of everything and tried again 5 times.

Another odd thing, when I have sparkrun setup do “auto” for the NCCL setup, it says “3 nodes 2 ports” and wants to do a Switch setup instead of a Ring setup.

I’m wondering if the QSFP cables could be the issue. I have 2 ASUS QSFP cables and 1 of the “Recommended by Nvidia” $99 cables I bought from nvidia.com store which basically seems like it is no name cable. It does not even say if it’s QSFP56 or QSFP112

Can someone show me what the diagnosis output of a 3 node cluster should look like?

So, I’m getting strange MTU’s on the CX7 links

user@spark1:~$ ip -br link | awk ‘{print $1}’ | while read i; do printf "%-15s " “$i”; ip link show “$i” | grep mtu; done
lo 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
enP7s7 2: enP7s7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
enp1s0f0np0 3: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
enp1s0f1np1 4: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
enP2p1s0f0np0 5: enP2p1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
enP2p1s0f1np1 6: enP2p1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
wlP9s9 7: wlP9s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
docker0 8: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default

user@spark2:~$ ip -br link | awk ‘{print $1}’ | while read i; do
printf "%-15s " “$i”
ip link show “$i” | grep mtu
done
lo 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
enP7s7 2: enP7s7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
enp1s0f0np0 3: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
enp1s0f1np1 4: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
enP2p1s0f0np0 5: enP2p1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
enP2p1s0f1np1 6: enP2p1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
wlP9s9 7: wlP9s9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
docker0 8: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
br-9be196cc0ce0 9: br-9be196cc0ce0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
br-b8171d04adf1 10: br-b8171d04adf1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
br-598a0d95bdb7 11: br-598a0d95bdb7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default

user@spark3:~$ ip -br link | awk ‘{print $1}’ | while read i; do
printf "%-15s " “$i”
ip link show “$i” | grep mtu
done
lo 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
enP7s7 2: enP7s7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
enp1s0f0np0 3: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
enp1s0f1np1 4: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
enP2p1s0f0np0 5: enP2p1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
enP2p1s0f1np1 6: enP2p1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
wlP9s9 7: wlP9s9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
docker0 8: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default

user@spark1:~$ for i in 192.168.10.223 192.168.0.1 192.168.2.2 192.168.1.1 192.168.3.2 192.168.10.225 192.168.0.225 192.168.2.225 192.168.1.225 192.168.3.225 192.168.10.235 192.168.0.235 192.168.2.235 192.168.1.235 192.168.3.235 ;do ssh $i “hostname” ;done
spark1
spark1
spark1
spark1
spark1
spark2
ssh: connect to host 192.168.0.225 port 22: No route to host
ssh: connect to host 192.168.2.225 port 22: No route to host
ssh: connect to host 192.168.1.225 port 22: No route to host
ssh: connect to host 192.168.3.225 port 22: No route to host
spark3
ssh: connect to host 192.168.0.235 port 22: No route to host
ssh: connect to host 192.168.2.235 port 22: No route to host
ssh: connect to host 192.168.1.235 port 22: No route to host
ssh: connect to host 192.168.3.235 port 22: No route to host

user@spark2:~$ for i in 192.168.10.223 192.168.0.1 192.168.2.2 192.168.1.1 192.168.3.2 192.168.10.225 192.168.0.225 192.168.2.225 192.168.1.225 192.168.3.225 192.168.10.235 192.168.0.235 192.168.2.235 192.168.1.235 192.168.3.235 ;do ssh $i “hostname” ;done
spark1
ssh: connect to host 192.168.0.1 port 22: No route to host
ssh: connect to host 192.168.2.2 port 22: No route to host
ssh: connect to host 192.168.1.1 port 22: No route to host
ssh: connect to host 192.168.3.2 port 22: No route to host
spark2
spark2
Warning: Permanently added ‘192.168.2.225’ (ED25519) to the list of known hosts.
spark2
spark2
Warning: Permanently added ‘192.168.3.225’ (ED25519) to the list of known hosts.
spark2
spark3
ssh: connect to host 192.168.0.235 port 22: No route to host
ssh: connect to host 192.168.2.235 port 22: No route to host
ssh: connect to host 192.168.1.235 port 22: No route to host
ssh: connect to host 192.168.3.235 port 22: No route to host

user@spark3:~$ for i in 192.168.10.223 192.168.0.1 192.168.2.2 192.168.1.1 192.168.3.2 192.168.10.225 192.168.0.225 192.168.2.225 192.168.1.225 192.168.3.225 192.168.10.235 192.168.0.235 192.168.2.235 192.168.1.235 192.168.3.235 ;do ssh $i “hostname” ;done
spark1
ssh: connect to host 192.168.0.1 port 22: No route to host
ssh: connect to host 192.168.2.2 port 22: No route to host
ssh: connect to host 192.168.1.1 port 22: No route to host
ssh: connect to host 192.168.3.2 port 22: No route to host
spark2
ssh: connect to host 192.168.0.225 port 22: No route to host
ssh: connect to host 192.168.2.225 port 22: No route to host
ssh: connect to host 192.168.1.225 port 22: No route to host
ssh: connect to host 192.168.3.225 port 22: No route to host
spark3
spark3
Warning: Permanently added ‘192.168.2.235’ (ED25519) to the list of known hosts.
spark3
spark3
Warning: Permanently added ‘192.168.3.235’ (ED25519) to the list of known hosts.
spark3

2

Have you tried explicitly configuring SSH? I have a cluster of 4 DGX Sparks and I have the configuration setup so that SSH always resolves the hostname to the correct IP. Based on what you posted here that’s my best guess as to what your problem is. The MTU should be 9000 (jumbo packets) for most switches I would think, but look at your own switch’s manual for more info on that.

3

Try this: