Why is my NCCL broken?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
21

To check if the 40-cx.yaml is error free, run sudo netplan try command. And using the example above there are no errors, so the netplan settings are okay:

elsaco@spark1:~$ sudo cat /etc/netplan/40-cx7.yaml
network:
  version: 2
  ethernets:
    enp1s0f0np0:
      addresses:
        - 192.168.100.11/24
      dhcp4: no
      dhcp6: no
      link-local: [ ipv4 ]
      mtu: 9000
    enP2p1s0f0np0:
      dhcp4: no
      dhcp6: no
      link-local: [ ipv4 ]
      mtu: 9000

elsaco@spark1:~$ sudo netplan try
Do you want to keep these settings?


Press ENTER before the timeout to accept the new configuration


Changes will revert in 113 seconds
Configuration accepted.

and interface setting using above config:

elsaco@spark1:~$ ip link show dev enp1s0f0np0
3: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 4c:bb:47:2d:a8:5d brd ff:ff:ff:ff:ff:ff

elsaco@spark1:~$ ip addr show dev enp1s0f0np0
3: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 4c:bb:47:2d:a8:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.11/24 brd 192.168.100.255 scope global noprefixroute enp1s0f0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::4ebb:47ff:fe2d:a85d/64 scope link
       valid_lft forever preferred_lft forever

The other end should use the same settings except the IP.

Sample mpirun:

elsaco@spark1:~$ mpirun -np 2 -H 192.168.100.11:1,192.168.100.22:1 --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH $HOME/nccl-tests/build/all_gather_perf
Warning: Permanently added '192.168.100.22' (ED25519) to the list of known hosts.
# nccl-tests version 2.17.6 nccl-headers=22803 nccl-library=22809
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   4206 on     spark1 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid   5343 on     spark2 device  0 [000f:01:00] NVIDIA GB10
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)
    33554432       4194304     float    none      -1   959.88   34.96   17.48       0   895.21   37.48   18.74       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 18.1098
#
# Collective test concluded: all_gather_perf

Sample ib_test_bw with Transport type: IB so settings looks okay:

elsaco@spark1:~$ ib_write_bw 192.168.100.22 -F --duration 10
 WARNING: BW peak won't be measured in this run.
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : rocep1s0f0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0158 PSN 0xf3684b RKey 0x182f00 VAddr 0x00e13a32f6d000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:11
 remote address: LID 0000 QPN 0x0158 PSN 0x66957e RKey 0x182f00 VAddr 0x00e13796b9d000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:100:22
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1246822          0.00               12987.74                   0.207804
---------------------------------------------------------------------------------------
22

Sorry, my instructions are incomplete.
You need to run this command on the receiving side (192.168.177.12):

ib_write_bw -d rocep1s0f0 --report_gbits -q 4 -R --force-link IB

I’ve updated the networking guide.

23

YES! That’s it, needed a respective listener at the other end. Thankyou @eurg, and everyone couldn’t have done it without you guys.

24

Yeah with the way your config files were set up before, you were only getting half the speed because you were only sending half of the port back and fourth.

When you configured your netplan files correctly, you opened up the full twin ports which basically doubled your speed and got you right to 200/GBs

25

enpXXX interfaces (used for TCP/IP comms) don’t matter for RDMA. You can have only one set up with an IP and be fine.

The trick was to specify both RoCE interfaces in export NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0

26

Ah yeah, that makes sense.