I have 4 nodes cluster , where 2 Nodes are connected with Switch-1 and other 2 nodes are connected with Switch-2.
- Spark-1 and Spark-2 are connected with Switch-1
- Spark-3 and Spark-4 are connected with Switch-2
- Each node ConnectX-7 NICs are connected with respective switch using 2x 100G QSFP.
- Inter-switch link is 400Gbps.
spark-dgx-01:~$
ip -4 -br a
enp1s0f0np0 UP 192.168.100.10/24
enP2p1s0f0np0 UP 192.168.101.10/24
ibv_devices
device node GUID
rocep1s0f0 4cbb4703002a7d61
rocep1s0f1 4cbb4703002a7d62
roceP2p1s0f0 4cbb4703002a7d65
roceP2p1s0f1 4cbb4703002a7d66
spark-dgx-02:~$
ip -4 -br a
enp1s0f0np0 UP 192.168.100.11/24
enP2p1s0f0np0 UP 192.168.101.11/24
ibv_devices
device node GUID
rocep1s0f0 4cbb4703002e9fe6
rocep1s0f1 4cbb4703002e9fe7
roceP2p1s0f0 4cbb4703002e9fea
roceP2p1s0f1 4cbb4703002e9feb
spark-dgx-03:~$
ip -4 -br a
enp1s0f0np0 UP 192.168.100.12/24
enP2p1s0f0np0 UP 192.168.101.12/24
ibv_devices
device node GUID
rocep1s0f0 4cbb4703002aab27
rocep1s0f1 4cbb4703002aab28
roceP2p1s0f0 4cbb4703002aab2b
roceP2p1s0f1 4cbb4703002aab2c
spark-dgx-04:~$ i
p -4 -br a
enp1s0f0np0 UP 192.168.100.13/24
enP2p1s0f0np0 UP 192.168.101.13/24
ibv_devices
device node GUID
rocep1s0f0 4cbb4703002a6531
rocep1s0f1 4cbb4703002a6532
roceP2p1s0f0 4cbb4703002a6535
roceP2p1s0f1 4cbb4703002a6536
NCCL performance results are significantly different when performed between nodes connected on single switch or nodes connected via 2 switches.
2 Nodes Connected on Same Switch
mpirun -np 2 -H 192.168.100.10:1,192.168.100.11:1 --mca plm_rsh_agent “ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -x” -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x UCX_NET_DEVICES -x NCCL_SOCKET_IFNAME -x OMPI_MCA_btl_tcp_if_include $HOME/nccl-tests/build/all_reduce_perf -b 512M -e 4G -f 2 -g 1 -n 100
Using devices
Rank 0 Group 0 Pid 661925 on spark-dgx-01 device 0 [000f:01:00] NVIDIA GB10
Rank 1 Group 0 Pid 33004 on spark-dgx-02 device 0 [000f:01:00] NVIDIA GB10
2 Node Connected on 2 Different Switches
mpirun -np 2 -H 192.168.100.10:1,192.168.100.12:1 --mca plm_rsh_agent “ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -x” -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x UCX_NET_DEVICES -x NCCL_SOCKET_IFNAME -x OMPI_MCA_btl_tcp_if_include $HOME/nccl-tests/build/all_reduce_perf -b 512M -e 4G -f 2 -g 1 -n 100
Using devices
Rank 0 Group 0 Pid 662315 on spark-dgx-01 device 0 [000f:01:00] NVIDIA GB10
Rank 1 Group 0 Pid 22369 on spark-dgx-03 device 0 [000f:01:00] NVIDIA GB10
out-of-place in-place
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
536870912 134217728 float sum -1 35964.8 14.93 14.93 0 35935.9 14.94 14.94 0
1073741824 268435456 float sum -1 70777.6 15.17 15.17 0 70867.0 15.15 15.15 0
2147483648 536870912 float sum -1 140334 15.30 15.30 0 140492 15.29 15.29 0
4294967296 1073741824 float sum -1 279325 15.38 15.38 0 279455 15.37 15.37 0
Out of bounds values : 0 OK
Avg bus bandwidth : 15.1904
Collective test concluded: all_reduce_perf
4 Nodes Connected on 2 Different Switches
mpirun -np 4 -H 192.168.100.10:1,192.168.100.11:1,192.168.100.12:1,192.168.100.13:1 --mca plm_rsh_agent “ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -x” -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x UCX_NET_DEVICES -x NCCL_SOCKET_IFNAME -x OMPI_MCA_btl_tcp_if_include $HOME/nccl-tests/build/all_reduce_perf -b 512M -e 4G -f 2 -g 1 -n 100
Using devices
Rank 0 Group 0 Pid 659081 on spark-dgx-01 device 0 [000f:01:00] NVIDIA GB10
Rank 1 Group 0 Pid 32625 on spark-dgx-02 device 0 [000f:01:00] NVIDIA GB10
Rank 2 Group 0 Pid 21978 on spark-dgx-03 device 0 [000f:01:00] NVIDIA GB10
Rank 3 Group 0 Pid 23895 on spark-dgx-04 device 0 [000f:01:00] NVIDIA GB10
out-of-place in-place
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
536870912 134217728 float sum -1 35964.8 14.93 14.93 0 35935.9 14.94 14.94 0
1073741824 268435456 float sum -1 70777.6 15.17 15.17 0 70867.0 15.15 15.15 0
2147483648 536870912 float sum -1 140334 15.30 15.30 0 140492 15.29 15.29 0
4294967296 1073741824 float sum -1 279325 15.38 15.38 0 279455 15.37 15.37 0
Out of bounds values : 0 OK
Avg bus bandwidth : 15.1904
Collective test concluded: all_reduce_perf