2x GB10 - nccl-tests won't build

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi all,
I just got a second ASUS GB10 and am trying to hook it together with my existing, running great one. Both systems were updated with the latest bios and latest OS/drivers/etc. One is completely out of the box clean with latest updates.

I am following the official docs on connecting 2 Sparks ( Connect Two Sparks | DGX Spark ) and that first part worked perfectly. They both see each other fine.

Then building NCCL ( NCCL for Multiple Sparks | DGX Spark ), I can build NCCL with Blackwell support fine, but get compile errors when I build step 3 - NCCL-Test, on both boxes I use the ‘Step 3’ instructions for building nccl-test.

Configs are standard Asus GX10 Ubuntu 24.04 reference install images with just standard apt update to latest of everything. Any clues?

...
make -C src build BUILDDIR=/home/nvidia/nccl-tests/build
make[1]: Entering directory '/home/nvidia/nccl-tests/src'
make[2]: Entering directory '/home/nvidia/nccl-tests/os'
make[2]: Leaving directory '/home/nvidia/nccl-tests/os'
Linking /home/nvidia/nccl-tests/build/device_api/gin/latency/ginGetLatency_ping_main.o > /home/nvidia/nccl-tests/build/device_api/gin/latency/ginGetLatency_ping_perf
/usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o: in function MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)': /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/op_inln.h:121:(.text._ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb[_ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb]+0x14): undefined reference to ompi_mpi_cxx_op_intercept'
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/op_inln.h:121:(.text._ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb[_ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb]+0x18): undefined reference to ompi_mpi_cxx_op_intercept' /usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o: in function MPI::Intracomm::Intracomm(ompi_communicator_t*)':
/usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23:(.text._ZNK3MPI9Intracomm5CloneEv[_ZNK3MPI9Intracomm5CloneEv]+0x40): undefined reference to MPI::Comm::Comm()' /usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o: in function MPI::Intracomm::Intracomm()':
/usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25:(.text._ZNK3MPI9Graphcomm5CloneEv[_ZNK3MPI9Graphcomm5CloneEv]+0x3c): undefined reference to MPI::Comm::Comm()' /usr/bin/ld: /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25:(.text._ZNK3MPI8Cartcomm3SubEPKb[_ZNK3MPI8Cartcomm3SubEPKb]+0xa0): undefined reference to MPI::Comm::Comm()'
/usr/bin/ld: /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25:(.text._ZNK3MPI9Intracomm12Create_graphEiPKiS2_b[_ZNK3MPI9Intracomm12Create_graphEiPKiS2_b]+0x3c): undefined reference to MPI::Comm::Comm()' /usr/bin/ld: /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25:(.text._ZNK3MPI8Cartcomm5CloneEv[_ZNK3MPI8Cartcomm5CloneEv]+0x3c): undefined reference to MPI::Comm::Comm()'
/usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o:/usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25: more undefined references to MPI::Comm::Comm()' follow /usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o:(.data.rel.ro._ZTVN3MPI8DatatypeE[_ZTVN3MPI8DatatypeE]+0x78): undefined reference to MPI::Datatype::Free()'
/usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o:(.data.rel.ro._ZTVN3MPI3WinE[_ZTVN3MPI3WinE]+0x48): undefined reference to `MPI::Win::Free()'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:246: /home/nvidia/nccl-tests/build/device_api/gin/latency/ginGetLatency_ping_perf] Error 1
make[1]: Leaving directory '/home/nvidia/nccl-tests/src'
make: *** [Makefile:20: src.build] Error 2

2

I’ve hit the same thing today. You need to edit ~/nccl-tests/src/Makefile and change this line:

NVLDFLAGS += -L$(MPI_HOME)/lib -L$(MPI_HOME)/lib64 -lmpi

to this line:

NVLDFLAGS += -L$(MPI_HOME)/lib -L$(MPI_HOME)/lib64 -lmpi -lmpi_cxx

(so, add -lmpi_cxx)

Then do:

cd ~/nccl-tests && make clean && make MPI=1

and it should build.

3

Awesome! That definitely fixed the compile problem! Then the next set of issues were:

  1. When running mpirun, got a missing lib so I had to fix that:
    /home/nvidia/nccl-tests/build/all_gather_perf: error while loading shared libraries: libnccl.so.2: cannot open shared object file: No such file or directory

Fixed the LD_LIBRARY_PATH to point to libnccl.so.2 I just built:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/nvidia/nccl/build/lib/

2. I re-ran mpirun and then got the spew of:

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

I fixed it by adding export HWLOC_COMPONENTS=-gl to my .bashrc as directed here:

https://forums.developer.nvidia.com/t/mpirun-authorization-required-but-no-authorization-protocol-specified/353580

  1. I then got my node1 and node2 management IP addresses that I set up with ssh by using:
    > ip addr show enP7s7
    I verified I could log in with ssh both directions without issues and both using the same nvidia username account I set up from part 1 of the setup.
    I also verified:
    nvidia@gx10-63e8:/$ ibdev2netdev
    rocep1s0f0 port 1 ==> enp1s0f0np0 (Down)
    rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)
    roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down)
    roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)

  2. I finally tried mpirun and got this error: 'internal error - please report this issue to the NCCL developers / '

mpirun -np 2 -H 10.0.0.42:1,10.0.0.61:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  $HOME/nccl-tests/build/all_gather_perf


mpirun -np 2 -H 10.0.0.42:1,10.0.0.61:1   --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"   -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH   $HOME/nccl-tests/build/all_gather_perf
Warning: Permanently added '10.0.0.61' (ED25519) to the list of known hosts.
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
#  Rank  0 Group  0 Pid  49269 on  gx10-63e8 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid  60250 on  localhost device  0 [000f:01:00] NVIDIA GB10
gx10-63e8: Test NCCL failure common.cu:236 'internal error - please report this issue to the NCCL developers / '
 .. gx10-63e8 pid 49269: Test failure common.cu:1712
 .. gx10-63e8 pid 49269: Test failure common.cu:1545
localhost: Test NCCL failure common.cu:236 'remote process exited or there was a network error / '
 .. localhost pid 60250: Test failure common.cu:1712
 .. localhost pid 60250: Test failure common.cu:1545
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Any ideas? Did the nccl-test binaries we built not have the right features?

4

Interesting, I ran with NCCL_DEBUG=INFO and it mentions not finding libnccl-env.so

nvidia@gx10-63e8:~/nccl-tests$ export NCCL_DEBUG=INFO
nvidia@gx10-63e8:~/nccl-tests$ mpirun -np 2 -H 10.0.0.42:1,10.0.0.61:1   --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"   -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH   $HOME/nccl-tests/build/all_gather_perf
Warning: Permanently added '10.0.0.61' (ED25519) to the list of known hosts.
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
#  Rank  0 Group  0 Pid  50757 on  gx10-63e8 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid  62459 on  localhost device  0 [000f:01:00] NVIDIA GB10
gx10-63e8:50757:50757 [0] NCCL INFO ncclOsDlopen(libnccl-env.so) failed: libnccl-env.so: cannot open shared object file: No such file or directory
gx10-63e8:50757:50757 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
gx10-63e8:50757:50757 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0

[2026-08-29 04:45:25] gx10-63e8:50757:50757 [0] bootstrap.cc:126 (bootstrapNetInit) NCCL WARN Bootstrap : no socket interface found
gx10-63e8:50757:50757 [0] NCCL INFO init.cc:151 (initOnceFunc) -> 5
gx10-63e8:50757:50757 [0] NCCL INFO init.cc:185 (ncclGetUniqueId) -> 5
gx10-63e8: Test NCCL failure common.cu:1641 'invalid usage (run with NCCL_DEBUG=WARN for details) / '
 .. gx10-63e8 pid 50757: Test failure common.cu:1545
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[2328,1],0]
  Exit code:    3
--------------------------------------------------------------------------
nvidia@gx10-63e8:~/nccl-tests$
5

I see that you are running mpirun on gx10-63e8. Is gx10-63e8 your head node or tail node?

From your output of mpirun:

# Using devices
#  Rank  0 Group  0 Pid  50757 on  gx10-63e8 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid  62459 on  localhost device  0 [000f:01:00] NVIDIA GB10

If gx10-63e8@10.0.0.42 is your head node, then I’m a bit confused by the fact that your tail node is on localhost.

What I did to my network settings is that I set static hostnames on both nodes and assigned static local IP addresses in /etc/hosts, so I can do something like this:

$ mpirun -np 2 -H GX10-00:1,GX10-01:1 hostname
GX10-00
GX10-01

$ ssh gx10-01 "type nvcc"
nvcc is /usr/local/cuda-13.2/bin/nvcc

Besides, instead of building NCCL without installing it, I built NCCL as packages and installed them system-wide. On Ubuntu/DGX OS, you can do something like:

$ # Install tools to create debian packages
$ sudo apt install build-essential devscripts debhelper fakeroot
$ # Build NCCL deb package
$ make -j20 pkg.debian.build NVCC_GENCODE="-gencode=arch=compute_121,code=sm_121"
$ ls build/pkg/deb/

and then install those deb packages using apt.

I’m also not a big fan of modifying LD_LIBRARY_PATH a lot. I just set CUDA_HOME, NCCL_HOME, and MPI_HOME on both nodes and never touched LD_LIBRARY_PATH manually. It’s a bit tedious because when sshing, only .bashrc is sourced (.bash_profile is not), so I need to export environment variables before the interactive-session-check in both nodes’ .bashrc.

After those steps, I can then run:

$ mpirun -np 2 -H GX10-00:1,GX10-01:1 \
  path/to/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
6

Thanks for the tips.

  1. I set the hostnames to gx10-00 and gx10-01 as you indicated adding to both /etc/hosts so they can talk to each other

$ mpirun -np 2 -H GX10-00:1,GX10-01:1 hostname
gx10-00
gx10-01
$ ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Down)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)

The cables are plugged into the rightmost plugs when looking directly at them from the rear.

  1. In order to get ssh gx10-01 “type nvcc” to work, I had to add these lines to the ~/.bashrc on both machines so it could find a path. Then nvcc reported correctly:
#fix ssh remote path issue
export PATH="/usr/local/cuda/bin:/opt/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"

export CUDA_HOME="/usr/local/cuda"
export MPI_HOME="/usr/lib/aarch64-linux-gnu/openmpi"
export NCCL_HOME="$HOME/nccl/build/"
export LD_LIBRARY_PATH="$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH"

export UCX_NET_DEVICES=enP7s7
export NCCL_SOCKET_IFNAME=enP7s7
export OMPI_MCA_btl_tcp_if_include=enP7s7
  1. Set up my 2 systems via ethernet cable with static ips: (10.0.0.50, 10.0.0.60) and their /etc/netplan/40-cx7.yaml on node 00:

network:
version: 2
ethernets:
enp1s0f0np0:
addresses:
- 192.168.50.11/24
dhcp4: no
enp1s0f1np1:
addresses:
- 192.168.60.13/24
dhcp4: no
enP2p1s0f0np0:
addresses:
- 192.168.50.15/24
dhcp4: no
enP2p1s0f1np1:
addresses:
- 192.168.60.17/24
dhcp4: no

Node 2:
network:
version: 2
ethernets:
enp1s0f0np0:
addresses:
- 192.168.50.10/24
dhcp4: no
enp1s0f1np1:
addresses:
- 192.168.60.12/24
dhcp4: no
enP2p1s0f0np0:
addresses:
- 192.168.50.14/24
dhcp4: no
enP2p1s0f1np1:
addresses:
- 192.168.60.16/24
dhcp4: no

ssh’ing between the two works great.

  1. I am able to test with ib_write_bd. Node 1 gets this to listen:
    $ ib_write_bw -d rocep1s0f1 --report_gbits -q 4 -R --force-link IB
    and Node 0 does:
$ ib_write_bw 192.168.60.12 -d rocep1s0f1 --report_gbits -q 4 -R --force-link IB
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : rocep1s0f1
 Number of qps   : 4		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x02b4 PSN 0x22c04e
 local address: LID 0000 QPN 0x02b5 PSN 0x415f20
 local address: LID 0000 QPN 0x02b6 PSN 0x3dc1ea
 local address: LID 0000 QPN 0x02b7 PSN 0x626bf1
 remote address: LID 0000 QPN 0x0230 PSN 0x8a90be
 remote address: LID 0000 QPN 0x0231 PSN 0xf9bed0
 remote address: LID 0000 QPN 0x0232 PSN 0xf69fda
 remote address: LID 0000 QPN 0x0233 PSN 0xbe8b21
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      20000            111.72             111.70 		   0.213049
---------------------------------------------------------------------------------------
  1. I built and installed the NCCL and installed the deb packages from build/pkg/deb/ as instructed
  2. When I run mpirun, I get this. I have also replaced it with the 10.0.0.50/60 ip addresses with the same result:
$ mpirun -np 2 -H GX10-00:1,GX10-01:1   /home/nvidia/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
#  Rank  0 Group  0 Pid  99247 on    gx10-00 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid 100615 on    gx10-01 device  0 [000f:01:00] NVIDIA GB10
gx10-00: Test NCCL failure common.cu:236 'internal error - please report this issue to the NCCL developers / '
 .. gx10-00 pid 99247: Test failure common.cu:1712
 .. gx10-00 pid 99247: Test failure common.cu:1545
gx10-01: Test NCCL failure common.cu:236 'remote process exited or there was a network error / '
 .. gx10-01 pid 100615: Test failure common.cu:1712
 .. gx10-01 pid 100615: Test failure common.cu:1545
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[15246,1],1]
  Exit code:    3
--------------------------------------------------------------------------
7

ib_write_bw works, so that’s good.

I also faced similar problems when trying to make nccl-tests to work, but it’s not the exact symptoms as yours, so I’m not sure if the following suggestions would help.

First, use ibv_devinfo -v to check if your static IP addresses are on the same GID. They probably are, since your ib_write_bw works.

Then, check if you have NCCL_IB_HCA environment variable set. If it’s automatically set to two ports like:

$ echo $NCCL_IB_HCA
rocep1s0f1,roceP2p1s0f1

or if it’s not set, then try setting it to use one port only:

$ export NCCL_IB_HCA=rocep1s0f1
$ # mpirun ...
$ export NCCL_IB_HCA=roceP2p1s0f1
$ # mpirun ...

If both ports work individually, then set NCCL_IB_HCA back to use two ports. In my case, I have to set it to use single port first, then to both ports, for nccl-tests to work properly. After it succeeded in using two ports, I don’t need to redo the above process after I reboot. I’m not exactly sure why that’s necessary.

Hope those help.

8

Interesting. I tried setting the NCCL_IB_HCA to each port individually on each machine, but same results. For the record, my plugs are in the far right hand connectors on both Node 00 and Node 01 machines.

When I ran ibv_devinfo -v, I get this on Node 00:
hca_id: rocep1s0f0
transport: InfiniBand (0)
fw_ver: 28.45.4028
node_guid: 30c5:9903:003f:63e9
sys_image_guid: 30c5:9903:003f:63e9

hca_id: rocep1s0f1
transport: InfiniBand (0)
fw_ver: 28.45.4028
node_guid: 30c5:9903:003f:bdc7
sys_image_guid: 30c5:9903:003f:bdc6

hca_id: roceP2p1s0f0
transport: InfiniBand (0)
fw_ver: 28.45.4028
node_guid: 30c5:9903:003f:bdca
sys_image_guid: 30c5:9903:003f:bdc6

hca_id: rocep1s0f1
transport: InfiniBand (0)
fw_ver: 28.45.4028
node_guid: 30c5:9903:003f:bdc7
sys_image_guid: 30c5:9903:003f:bdc6

And this when I run it on Node 01:
hca_id: rocep1s0f0
transport: InfiniBand (0)
fw_ver: 28.45.4028
node_guid: 30c5:9903:003f:bdc6
sys_image_guid: 30c5:9903:003f:bdc6

hca_id: rocep1s0f1
transport: InfiniBand (0)
fw_ver: 28.45.4028
node_guid: 30c5:9903:003f:63ea
sys_image_guid: 30c5:9903:003f:63e9

hca_id: roceP2p1s0f0
transport: InfiniBand (0)
fw_ver: 28.45.4028
node_guid: 30c5:9903:003f:63ed
sys_image_guid: 30c5:9903:003f:63e9

hca_id: roceP2p1s0f1
transport: InfiniBand (0)
fw_ver: 28.45.4028
node_guid: 30c5:9903:003f:63ee
sys_image_guid: 30c5:9903:003f:63e9

Should they be different or match?

9

On different nodes, guid should differ. Within one node, node_guids are different and sys_image_guid s are the same. Yours seem right to me (you probably mistakenly swapped rocep1s0f0 info)

Have you checked the GID? You can do so by doing something like:

$ mpirun -np 2 -H GX10-00:1,GX10-01:1 ibv_devinfo -v | grep 192.
                GID[  3]:               ::ffff:192.168.100.10, RoCE v2
                GID[  3]:               ::ffff:192.168.100.11, RoCE v2
                GID[  3]:               ::ffff:192.168.101.10, RoCE v2
                GID[  3]:               ::ffff:192.168.101.11, RoCE v2

I think the local IP addresses need to be on the same GID (in my case, GID[3]).

10

$ mpirun -np 2 -H GX10-00:1,GX10-01:1 ibv_devinfo -v | grep 192.
GID[ 3]: ::ffff:192.168.60.12, RoCE v2
GID[ 3]: ::ffff:192.168.60.13, RoCE v2
GID[ 3]: ::ffff:192.168.60.16, RoCE v2
GID[ 3]: ::ffff:192.168.60.17, RoCE v2

11

I’m out of clues, then. Sorry buddy :(

After installing NCCL as system-wide packages, you don’t really need to export anything to LD_LIBRARY_PATH (just make sure NCCL_HOME points to the system-wide installation path) since it should be accessible system-wide, but I don’t see the reason why your approach would fail.

When you set NCCL_DEBUG=INFO, are you still seeing the complaint about NCCL-related .so files?

Lastly, maybe try compiling nccl-tests v2.19.7 to see if that works? v2.20.0 introduced some changes that break compilation on my end (because I’m running Fedora 44 instead of DGX OS/Ubuntu), so I cannot immediately verify if v2.20.0 works on my end (I’m on commit 43cb65e).

12

Remove the 200G link in port 0 and try again. There is something weird going on with NCCL.

Also you really don’t need two 200G links between two nodes, and from the way the CX-7 is linked back to the GPU/CPU you only have 256G worth of bandwidth anyways.

13

Ok - the solution was stranger than I would like. In the end, I went back to my standard Ubuntu user account I created when I set up the Asus GX10 box (‘me’). If I used the official nvidia setup script ( Connect Two Sparks | DGX Spark ) to create an ‘nvidia’ account the way it says, the account doesn’t seem to pick up any of the standard paths/scripting/setup as a ‘normal’ account.

Instead, I used my accounts that I set up when I re-imaged the system. I followed the script again and didn’t need to do any LD_PATH fixes or anything else - it just worked:

me@gx10-00:~$ ib_write_bw 192.168.100.11 -d rocep1s0f1 --report_gbits -q 4 -R --force-link IB
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : rocep1s0f1
 Number of qps   : 4		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 1024[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs	 : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0349 PSN 0x55da7b
 local address: LID 0000 QPN 0x034a PSN 0x3f249
 local address: LID 0000 QPN 0x034b PSN 0x19405f
 local address: LID 0000 QPN 0x034c PSN 0x5061c2
 remote address: LID 0000 QPN 0x0349 PSN 0x4262d1
 remote address: LID 0000 QPN 0x034a PSN 0x7f42e7
 remote address: LID 0000 QPN 0x034b PSN 0xb087a5
 remote address: LID 0000 QPN 0x034c PSN 0x4a9e90
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      20000            111.72             111.70 		   0.213057
---------------------------------------------------------------------------------------

and then:

me@gx10-00:~$ mpirun -np 2 -H 10.0.0.50:1,10.0.0.60:1   --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"   -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH   $HOME/nccl-tests/build/all_gather_perf
Invalid MIT-MAGIC-COOKIE-1 key
Invalid MIT-MAGIC-COOKIE-1 key
Warning: Permanently added '10.0.0.60' (ED25519) to the list of known hosts.
Invalid MIT-MAGIC-COOKIE-1 key
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
#  Rank  0 Group  0 Pid  49848 on    gx10-00 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid  62485 on    gx10-01 device  0 [000f:01:00] NVIDIA GB10
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)        
    33554432       4194304     float    none      -1  1106.13   30.33   15.17       0  2315.56   14.49    7.25       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 11.2064 
#
# Collective test concluded: all_gather_perf
#

So, the only remaining question is - are those GB/s numbers correct for 2 GX10’s connected with a single cable?

14

After you installed DACs in the CX7 you need to power off and unplug the GB10’s for a minute as the NIC needs a full power drain for the discovered settings to set the very first time.

15

200 Gb/s is 25 GB/s so your busbws are a bit off. Maybe it’s because you’re running a very small test - try path/to/all_gather_perf -b 16G -e 16G -f 2 instead. For busbw, you should get something around 23~24 GB/s.

16

After you installed DACs in the CX7 you need to power off and unplug the GB10’s for a minute as the NIC needs a full power drain for the discovered settings to set the very first time.

Interesting. I didn’t seem to have problems with this - but I rebooted so many times and they had been connected for 2 days at that point so maybe it wasn’t something I ran into.

17

It doesn’t happen to everyone but it is very common, as in a post or two a week on these forums.

18

Here’s what I get when running that command:

$ mpirun -np 2 -H 10.0.0.50:1,10.0.0.60:1   --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"   -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH   $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
Warning: Permanently added '10.0.0.60' (ED25519) to the list of known hosts.
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
#  Rank  0 Group  0 Pid 1363361 on    gx10-00 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid 1353266 on    gx10-01 device  0 [000f:01:00] NVIDIA GB10
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)        
 17179869184    2147483648     float    none      -1   519884   33.05   16.52       0   480074   35.79   17.89       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 17.2079 
#
# Collective test concluded: all_gather_perf
#

Looks like i’m still below par. Is this a setting issue or possibly a bad cable/port?

I was able to get eugr/spark-vllm-docker up and going with a wide variety of the major models (deepseek v4, qwen, etc). They seem to all work.

19

If your inference throughput is as expected, then I’d say you are good to go. Debugging those tests does get annoying, and if real-world usage isn’t affected, then I won’t bother fixing them.

Your test results do look weird, though. ~17 GB/s doesn’t really make sense, because it’s much lower than 25 GB/s but also much higher than 12.5 GB/s. I think a lower-grade QSFP cable (QSFP28) goes up to 100 Gbps, so it’s hard to tell where ~17 GB/s comes from.

Your ib_write_bw result shows around 111 Gbps on a single port, which is also weird. I think each logic port is capped at 100 Gbps, and my single-port BW average is around 99 Gbps.

Still, if you are happy with your inference throughput, then those numbers probably don’t really matter.