# 2x GB10 - nccl-tests won't build

**URL:** <https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746>\
**Category:** DGX Spark / GB10\
**Created:** [August 28, 2026, 10:09pm UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746 "2026-08-28T22:09:45Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 28, 2026, 10:09pm UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/1 "2026-08-28T22:09:45Z")

</div>

Hi all,  
I just got a second ASUS GB10 and am trying to hook it together with my existing, running great one. Both systems were updated with the latest bios and latest OS/drivers/etc. One is completely out of the box clean with latest updates.

I am following the official docs on connecting 2 Sparks ( [Connect Two Sparks | DGX Spark](https://build.nvidia.com/spark/connect-two-sparks) ) and that first part worked perfectly. They both see each other fine.

Then building NCCL ( [NCCL for Multiple Sparks | DGX Spark](https://build.nvidia.com/spark/nccl/stacked-sparks) ), I can build NCCL with Blackwell support fine, but get compile errors when I build step 3 - NCCL-Test, on both boxes I use the ‘Step 3’ instructions for building nccl-test.

Configs are standard Asus GX10 Ubuntu 24.04 reference install images with just standard apt update to latest of everything. Any clues?

`...`  
`make -C src build BUILDDIR=/home/nvidia/nccl-tests/build`  
`make[1]: Entering directory '/home/nvidia/nccl-tests/src'`  
`make[2]: Entering directory '/home/nvidia/nccl-tests/os'`  
`make[2]: Leaving directory '/home/nvidia/nccl-tests/os'`  
`Linking /home/nvidia/nccl-tests/build/device_api/gin/latency/ginGetLatency_ping_main.o > /home/nvidia/nccl-tests/build/device_api/gin/latency/ginGetLatency_ping_perf`  
`/usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o: in function MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)': /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/op_inln.h:121:(.text._ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb[_ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb]+0x14): undefined reference to ompi_mpi_cxx_op_intercept'`  
`/usr/bin/ld: /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/op_inln.h:121:(.text._ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb[_ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb]+0x18): undefined reference to ompi_mpi_cxx_op_intercept' /usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o: in function MPI::Intracomm::Intracomm(ompi_communicator_t*)':`  
`/usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23:(.text._ZNK3MPI9Intracomm5CloneEv[_ZNK3MPI9Intracomm5CloneEv]+0x40): undefined reference to MPI::Comm::Comm()' /usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o: in function MPI::Intracomm::Intracomm()':`  
`/usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25:(.text._ZNK3MPI9Graphcomm5CloneEv[_ZNK3MPI9Graphcomm5CloneEv]+0x3c): undefined reference to MPI::Comm::Comm()' /usr/bin/ld: /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25:(.text._ZNK3MPI8Cartcomm3SubEPKb[_ZNK3MPI8Cartcomm3SubEPKb]+0xa0): undefined reference to MPI::Comm::Comm()'`  
`/usr/bin/ld: /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25:(.text._ZNK3MPI9Intracomm12Create_graphEiPKiS2_b[_ZNK3MPI9Intracomm12Create_graphEiPKiS2_b]+0x3c): undefined reference to MPI::Comm::Comm()' /usr/bin/ld: /usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25:(.text._ZNK3MPI8Cartcomm5CloneEv[_ZNK3MPI8Cartcomm5CloneEv]+0x3c): undefined reference to MPI::Comm::Comm()'`  
`/usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o:/usr/lib/aarch64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h:25: more undefined references to MPI::Comm::Comm()' follow /usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o:(.data.rel.ro._ZTVN3MPI8DatatypeE[_ZTVN3MPI8DatatypeE]+0x78): undefined reference to MPI::Datatype::Free()'`  
`/usr/bin/ld: /home/nvidia/nccl-tests/build/device_api/gin/utils/common.o:(.data.rel.ro._ZTVN3MPI3WinE[_ZTVN3MPI3WinE]+0x48): undefined reference to `MPI::Win::Free()'`  
`collect2: error: ld returned 1 exit status`  
`make[1]: *** [Makefile:246: /home/nvidia/nccl-tests/build/device_api/gin/latency/ginGetLatency_ping_perf] Error 1`  
`make[1]: Leaving directory '/home/nvidia/nccl-tests/src'`  
`make: *** [Makefile:20: src.build] Error 2`

---

<div class="post-metadata">

**Author:** ![kamil.andrusz](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@kamil.andrusz](https://forums.developer.nvidia.com/u/kamil.andrusz)\
**Post date:** [August 28, 2026, 11:50pm UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/2 "2026-08-28T23:50:50Z")

</div>

I’ve hit the same thing today. You need to edit `~/nccl-tests/src/Makefile` and change this line:

`NVLDFLAGS += -L$(MPI_HOME)/lib -L$(MPI_HOME)/lib64 -lmpi`

to this line:

`NVLDFLAGS += -L$(MPI_HOME)/lib -L$(MPI_HOME)/lib64 -lmpi -lmpi_cxx`

(so, add `-lmpi_cxx`)

Then do:

`cd ~/nccl-tests && make clean && make MPI=1`

and it should build.

---

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 29, 2026, 4:27am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/3 "2026-08-29T04:27:24Z")

</div>

Awesome! That definitely fixed the compile problem! Then the next set of issues were:

1. When running mpirun, got a missing lib so I had to fix that:  
`/home/nvidia/nccl-tests/build/all_gather_perf: error while loading shared libraries: libnccl.so.2: cannot open shared object file: No such file or directory`

Fixed the LD\_LIBRARY\_PATH to point to libnccl.so.2 I just built:  
`export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/nvidia/nccl/build/lib/`

2. I re-ran mpirun and then got the spew of:

```auto
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

```

I fixed it by adding `export HWLOC_COMPONENTS=-gl` to my .bashrc as directed here:

[https://forums.developer.nvidia.com/t/mpirun-authorization-required-but-no-authorization-protocol-specified/353580](https://forums.developer.nvidia.com/t/mpirun-authorization-required-but-no-authorization-protocol-specified/353580)

1. I then got my node1 and node2 management IP addresses that I set up with ssh by using:  
`> ip addr show enP7s7`  
I verified I could log in with ssh both directions without issues and both using the same nvidia username account I set up from part 1 of the setup.  
I also verified:  
`nvidia@gx10-63e8:/$ ibdev2netdev`  
`rocep1s0f0 port 1 ==> enp1s0f0np0 (Down)`  
`rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)`  
`roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down)`  
`roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)`

2. I finally tried mpirun and got this error: `'internal error - please report this issue to the NCCL developers / '`

```auto
mpirun -np 2 -H 10.0.0.42:1,10.0.0.61:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  $HOME/nccl-tests/build/all_gather_perf

mpirun -np 2 -H 10.0.0.42:1,10.0.0.61:1 --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH $HOME/nccl-tests/build/all_gather_perf
Warning: Permanently added '10.0.0.61' (ED25519) to the list of known hosts.
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
# Rank 0 Group 0 Pid 49269 on gx10-63e8 device 0 [000f:01:00] NVIDIA GB10
# Rank 1 Group 0 Pid 60250 on localhost device 0 [000f:01:00] NVIDIA GB10
gx10-63e8: Test NCCL failure common.cu:236 'internal error - please report this issue to the NCCL developers / '
 .. gx10-63e8 pid 49269: Test failure common.cu:1712
 .. gx10-63e8 pid 49269: Test failure common.cu:1545
localhost: Test NCCL failure common.cu:236 'remote process exited or there was a network error / '
 .. localhost pid 60250: Test failure common.cu:1712
 .. localhost pid 60250: Test failure common.cu:1545
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

```

Any ideas? Did the nccl-test binaries we built not have the right features?

---

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 29, 2026, 4:55am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/4 "2026-08-29T04:55:35Z")

</div>

Interesting, I ran with NCCL\_DEBUG=INFO and it mentions not finding libnccl-env.so

```auto
nvidia@gx10-63e8:~/nccl-tests$ export NCCL_DEBUG=INFO
nvidia@gx10-63e8:~/nccl-tests$ mpirun -np 2 -H 10.0.0.42:1,10.0.0.61:1 --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH $HOME/nccl-tests/build/all_gather_perf
Warning: Permanently added '10.0.0.61' (ED25519) to the list of known hosts.
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
# Rank 0 Group 0 Pid 50757 on gx10-63e8 device 0 [000f:01:00] NVIDIA GB10
# Rank 1 Group 0 Pid 62459 on localhost device 0 [000f:01:00] NVIDIA GB10
gx10-63e8:50757:50757 [0] NCCL INFO ncclOsDlopen(libnccl-env.so) failed: libnccl-env.so: cannot open shared object file: No such file or directory
gx10-63e8:50757:50757 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so
gx10-63e8:50757:50757 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0

[2026-08-29 04:45:25] gx10-63e8:50757:50757 [0] bootstrap.cc:126 (bootstrapNetInit) NCCL WARN Bootstrap : no socket interface found
gx10-63e8:50757:50757 [0] NCCL INFO init.cc:151 (initOnceFunc) -> 5
gx10-63e8:50757:50757 [0] NCCL INFO init.cc:185 (ncclGetUniqueId) -> 5
gx10-63e8: Test NCCL failure common.cu:1641 'invalid usage (run with NCCL_DEBUG=WARN for details) / '
 .. gx10-63e8 pid 50757: Test failure common.cu:1545
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[2328,1],0]
  Exit code: 3
--------------------------------------------------------------------------
nvidia@gx10-63e8:~/nccl-tests$

```

---

<div class="post-metadata">

**Author:** ![hiroshiya](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hiroshiya/32/528526_2.png) [@hiroshiya](https://forums.developer.nvidia.com/u/hiroshiya)\
**Post date:** [August 29, 2026, 7:33am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/5 "2026-08-29T07:33:00Z")

</div>

I see that you are running `mpirun` on `gx10-63e8`. Is `gx10-63e8` your head node or tail node?

From your output of `mpirun`:

```plaintext
# Using devices
# Rank 0 Group 0 Pid 50757 on gx10-63e8 device 0 [000f:01:00] NVIDIA GB10
# Rank 1 Group 0 Pid 62459 on localhost device 0 [000f:01:00] NVIDIA GB10

```

If `gx10-63e8@10.0.0.42` is your head node, then I’m a bit confused by the fact that your tail node is on `localhost`.

What I did to my network settings is that I set static hostnames on both nodes and assigned static local IP addresses in `/etc/hosts`, so I can do something like this:

```bash
$ mpirun -np 2 -H GX10-00:1,GX10-01:1 hostname
GX10-00
GX10-01

$ ssh gx10-01 "type nvcc"
nvcc is /usr/local/cuda-13.2/bin/nvcc

```

Besides, instead of building `NCCL` without installing it, I built `NCCL` as packages and installed them system-wide. On Ubuntu/DGX OS, you can do something like:

```bash
$ # Install tools to create debian packages
$ sudo apt install build-essential devscripts debhelper fakeroot
$ # Build NCCL deb package
$ make -j20 pkg.debian.build NVCC_GENCODE="-gencode=arch=compute_121,code=sm_121"
$ ls build/pkg/deb/

```

and then install those `deb` packages using `apt`.

I’m also not a big fan of modifying `LD_LIBRARY_PATH` a lot. I just set `CUDA_HOME`, `NCCL_HOME`, and `MPI_HOME` on both nodes and never touched `LD_LIBRARY_PATH` manually. It’s a bit tedious because when `ssh`ing, only `.bashrc` is sourced (`.bash_profile` is not), so I need to export environment variables before the interactive-session-check in both nodes’ `.bashrc`.

After those steps, I can then run:

```bash
$ mpirun -np 2 -H GX10-00:1,GX10-01:1 \
  path/to/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2

```

---

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 30, 2026, 1:07am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/6 "2026-08-30T01:07:10Z")

</div>

> [@hiroshiya](#):
>
> ```auto
> mpirun -np 2 -H GX10-00:1,GX10-01:1 \
> path/to/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
> 
> ```

Thanks for the tips.

1. I set the hostnames to gx10-00 and gx10-01 as you indicated adding to both /etc/hosts so they can talk to each other

`$ mpirun -np 2 -H GX10-00:1,GX10-01:1 hostname`  
`gx10-00`  
`gx10-01`  
`$ ibdev2netdev`  
`rocep1s0f0 port 1 ==> enp1s0f0np0 (Down)`  
`rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)`  
`roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down)`  
`roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)`

The cables are plugged into the rightmost plugs when looking directly at them from the rear.

1. In order to get ssh gx10-01 “type nvcc” to work, I had to add these lines to the ~/.bashrc on both machines so it could find a path. Then nvcc reported correctly:

```auto
#fix ssh remote path issue
export PATH="/usr/local/cuda/bin:/opt/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"

export CUDA_HOME="/usr/local/cuda"
export MPI_HOME="/usr/lib/aarch64-linux-gnu/openmpi"
export NCCL_HOME="$HOME/nccl/build/"
export LD_LIBRARY_PATH="$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH"

export UCX_NET_DEVICES=enP7s7
export NCCL_SOCKET_IFNAME=enP7s7
export OMPI_MCA_btl_tcp_if_include=enP7s7

```

1. Set up my 2 systems via ethernet cable with static ips: (10.0.0.50, 10.0.0.60) and their /etc/netplan/40-cx7.yaml on node 00:

`network:`  
`version: 2`  
`ethernets:`  
`enp1s0f0np0:`  
`addresses:`  
`- 192.168.50.11/24`  
`dhcp4: no`  
`enp1s0f1np1:`  
`addresses:`  
`- 192.168.60.13/24`  
`dhcp4: no`  
`enP2p1s0f0np0:`  
`addresses:`  
`- 192.168.50.15/24`  
`dhcp4: no`  
`enP2p1s0f1np1:`  
`addresses:`  
`- 192.168.60.17/24`  
`dhcp4: no`

Node 2:  
`network:`  
`version: 2`  
`ethernets:`  
`enp1s0f0np0:`  
`addresses:`  
`- 192.168.50.10/24`  
`dhcp4: no`  
`enp1s0f1np1:`  
`addresses:`  
`- 192.168.60.12/24`  
`dhcp4: no`  
`enP2p1s0f0np0:`  
`addresses:`  
`- 192.168.50.14/24`  
`dhcp4: no`  
`enP2p1s0f1np1:`  
`addresses:`  
`- 192.168.60.16/24`  
`dhcp4: no`

ssh’ing between the two works great.

1. I am able to test with ib\_write\_bd. Node 1 gets this to listen:  
$ ib\_write\_bw -d rocep1s0f1 --report\_gbits -q 4 -R --force-link IB  
and Node 0 does:

```auto
$ ib_write_bw 192.168.60.12 -d rocep1s0f1 --report_gbits -q 4 -R --force-link IB
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port : OFF Device : rocep1s0f1
 Number of qps : 4 Transport type : IB
 Connection type : RC Using SRQ : OFF
 PCIe relax order: ON
 ibv_wr* API : ON
 TX depth : 128
 CQ Moderation : 1
 Mtu : 1024[B]
 Link type : IB
 Max inline data : 0[B]
 rdma_cm QPs : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x02b4 PSN 0x22c04e
 local address: LID 0000 QPN 0x02b5 PSN 0x415f20
 local address: LID 0000 QPN 0x02b6 PSN 0x3dc1ea
 local address: LID 0000 QPN 0x02b7 PSN 0x626bf1
 remote address: LID 0000 QPN 0x0230 PSN 0x8a90be
 remote address: LID 0000 QPN 0x0231 PSN 0xf9bed0
 remote address: LID 0000 QPN 0x0232 PSN 0xf69fda
 remote address: LID 0000 QPN 0x0233 PSN 0xbe8b21
---------------------------------------------------------------------------------------
 #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
 65536 20000 111.72 111.70 0.213049
---------------------------------------------------------------------------------------

```

1. I built and installed the NCCL and installed the deb packages from build/pkg/deb/ as instructed
2. When I run mpirun, I get this. I have also replaced it with the 10.0.0.50/60 ip addresses with the same result:

```auto
$ mpirun -np 2 -H GX10-00:1,GX10-01:1 /home/nvidia/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
# Rank 0 Group 0 Pid 99247 on gx10-00 device 0 [000f:01:00] NVIDIA GB10
# Rank 1 Group 0 Pid 100615 on gx10-01 device 0 [000f:01:00] NVIDIA GB10
gx10-00: Test NCCL failure common.cu:236 'internal error - please report this issue to the NCCL developers / '
 .. gx10-00 pid 99247: Test failure common.cu:1712
 .. gx10-00 pid 99247: Test failure common.cu:1545
gx10-01: Test NCCL failure common.cu:236 'remote process exited or there was a network error / '
 .. gx10-01 pid 100615: Test failure common.cu:1712
 .. gx10-01 pid 100615: Test failure common.cu:1545
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[15246,1],1]
  Exit code: 3
--------------------------------------------------------------------------

```

---

<div class="post-metadata">

**Author:** ![hiroshiya](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hiroshiya/32/528526_2.png) [@hiroshiya](https://forums.developer.nvidia.com/u/hiroshiya)\
**Post date:** [August 30, 2026, 2:10am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/7 "2026-08-30T02:10:27Z")

</div>

`ib_write_bw` works, so that’s good.

I also faced similar problems when trying to make `nccl-tests` to work, but it’s not the exact symptoms as yours, so I’m not sure if the following suggestions would help.

First, use `ibv_devinfo -v` to check if your static IP addresses are on the same GID. They probably are, since your `ib_write_bw` works.

Then, check if you have `NCCL_IB_HCA` environment variable set. If it’s automatically set to two ports like:

```bash
$ echo $NCCL_IB_HCA
rocep1s0f1,roceP2p1s0f1

```

or if it’s not set, then try setting it to use one port only:

```bash
$ export NCCL_IB_HCA=rocep1s0f1
$ # mpirun ...
$ export NCCL_IB_HCA=roceP2p1s0f1
$ # mpirun ...

```

If both ports work individually, then set `NCCL_IB_HCA` back to use two ports. In my case, I have to set it to use single port first, then to both ports, for `nccl-tests` to work properly. After it succeeded in using two ports, I don’t need to redo the above process after I reboot. I’m not exactly sure why that’s necessary.

Hope those help.

---

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 30, 2026, 2:53am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/8 "2026-08-30T02:53:35Z")

</div>

> [@hiroshiya](#):
>
> ibv\_devinfo -v

Interesting. I tried setting the NCCL\_IB\_HCA to each port individually on each machine, but same results. For the record, my plugs are in the far right hand connectors on both Node 00 and Node 01 machines.

**When I ran ibv\_devinfo -v, I get this on Node 00:**  
`hca_id:	rocep1s0f0`  
`transport: InfiniBand (0)`  
`fw_ver: 28.45.4028`  
`node_guid: 30c5:9903:003f:63e9`  
`sys_image_guid: 30c5:9903:003f:63e9`

`hca_id:	rocep1s0f1`  
`transport: InfiniBand (0)`  
`fw_ver: 28.45.4028`  
`node_guid: 30c5:9903:003f:bdc7`  
`sys_image_guid: 30c5:9903:003f:bdc6`

`hca_id:	roceP2p1s0f0`  
`transport: InfiniBand (0)`  
`fw_ver: 28.45.4028`  
`node_guid: 30c5:9903:003f:bdca`  
`sys_image_guid: 30c5:9903:003f:bdc6`

`hca_id:	rocep1s0f1`  
`transport: InfiniBand (0)`  
`fw_ver: 28.45.4028`  
`node_guid: 30c5:9903:003f:bdc7`  
`sys_image_guid: 30c5:9903:003f:bdc6`

**And this when I run it on Node 01:**  
`hca_id:	rocep1s0f0`  
`transport: InfiniBand (0)`  
`fw_ver: 28.45.4028`  
`node_guid: 30c5:9903:003f:bdc6`  
`sys_image_guid: 30c5:9903:003f:bdc6`

`hca_id:	rocep1s0f1`  
`transport: InfiniBand (0)`  
`fw_ver: 28.45.4028`  
`node_guid: 30c5:9903:003f:63ea`  
`sys_image_guid: 30c5:9903:003f:63e9`

`hca_id:	roceP2p1s0f0`  
`transport: InfiniBand (0)`  
`fw_ver: 28.45.4028`  
`node_guid: 30c5:9903:003f:63ed`  
`sys_image_guid: 30c5:9903:003f:63e9`

`hca_id:	roceP2p1s0f1`  
`transport: InfiniBand (0)`  
`fw_ver: 28.45.4028`  
`node_guid: 30c5:9903:003f:63ee`  
`sys_image_guid: 30c5:9903:003f:63e9`

Should they be different or match?

---

<div class="post-metadata">

**Author:** ![hiroshiya](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hiroshiya/32/528526_2.png) [@hiroshiya](https://forums.developer.nvidia.com/u/hiroshiya)\
**Post date:** [August 30, 2026, 3:15am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/9 "2026-08-30T03:15:31Z")

</div>

On different nodes, `guid` should differ. Within one node, `node_guid`s are different and `sys_image_guid` s are the same. Yours seem right to me (you probably mistakenly swapped `rocep1s0f0` info)

Have you checked the `GID`? You can do so by doing something like:

```bash
$ mpirun -np 2 -H GX10-00:1,GX10-01:1 ibv_devinfo -v | grep 192.
                GID[3]: ::ffff:192.168.100.10, RoCE v2
                GID[3]: ::ffff:192.168.100.11, RoCE v2
                GID[3]: ::ffff:192.168.101.10, RoCE v2
                GID[3]: ::ffff:192.168.101.11, RoCE v2

```

I think the local IP addresses need to be on the same GID (in my case, `GID[3]`).

---

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 30, 2026, 5:36am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/10 "2026-08-30T05:36:22Z")

</div>

> [@hiroshiya](#):
>
> `mpirun -np 2 -H GX10-00:1,GX10-01:1 ibv_devinfo -v | grep 192.`

$ mpirun -np 2 -H GX10-00:1,GX10-01:1 ibv\_devinfo -v | grep 192.  
GID[3]: ::ffff:192.168.60.12, RoCE v2  
GID[3]: ::ffff:192.168.60.13, RoCE v2  
GID[3]: ::ffff:192.168.60.16, RoCE v2  
GID[3]: ::ffff:192.168.60.17, RoCE v2

---

<div class="post-metadata">

**Author:** ![hiroshiya](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hiroshiya/32/528526_2.png) [@hiroshiya](https://forums.developer.nvidia.com/u/hiroshiya)\
**Post date:** [August 30, 2026, 6:14am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/11 "2026-08-30T06:14:43Z")

</div>

I’m out of clues, then. Sorry buddy :(

After installing NCCL as system-wide packages, you don’t really need to export anything to `LD_LIBRARY_PATH` (just make sure `NCCL_HOME` points to the system-wide installation path) since it should be accessible system-wide, but I don’t see the reason why your approach would fail.

When you set `NCCL_DEBUG=INFO`, are you still seeing the complaint about NCCL-related `.so` files?

Lastly, maybe try compiling `nccl-tests v2.19.7` to see if that works? `v2.20.0` introduced some changes that break compilation on my end (because I’m running Fedora 44 instead of DGX OS/Ubuntu), so I cannot immediately verify if `v2.20.0` works on my end (I’m on commit `43cb65e`).

---

<div class="post-metadata">

**Author:** ![mashie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mashie/32/503632_2.png) [@mashie](https://forums.developer.nvidia.com/u/mashie)\
**Post date:** [August 30, 2026, 9:52am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/12 "2026-08-30T09:52:34Z")

</div>

Remove the 200G link in port 0 and try again. There is something weird going on with NCCL.

Also you really don’t need two 200G links between two nodes, and from the way the CX-7 is linked back to the GPU/CPU you only have 256G worth of bandwidth anyways.

---

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 30, 2026, 6:46pm UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/13 "2026-08-30T18:46:23Z")

</div>

Ok - the solution was stranger than I would like. In the end, I went back to my standard Ubuntu user account I created when I set up the Asus GX10 box (‘me’). If I used the official nvidia setup script ( [Connect Two Sparks | DGX Spark](https://build.nvidia.com/spark/connect-two-sparks/stacked-sparks) ) to create an ‘nvidia’ account the way it says, the account doesn’t seem to pick up any of the standard paths/scripting/setup as a ‘normal’ account.

Instead, I used my accounts that I set up when I re-imaged the system. I followed the script again and didn’t need to do any LD\_PATH fixes or anything else - it just worked:

```auto
me@gx10-00:~$ ib_write_bw 192.168.100.11 -d rocep1s0f1 --report_gbits -q 4 -R --force-link IB
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port : OFF Device : rocep1s0f1
 Number of qps : 4 Transport type : IB
 Connection type : RC Using SRQ : OFF
 PCIe relax order: ON
 ibv_wr* API : ON
 TX depth : 128
 CQ Moderation : 1
 Mtu : 1024[B]
 Link type : IB
 Max inline data : 0[B]
 rdma_cm QPs : ON
 Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0349 PSN 0x55da7b
 local address: LID 0000 QPN 0x034a PSN 0x3f249
 local address: LID 0000 QPN 0x034b PSN 0x19405f
 local address: LID 0000 QPN 0x034c PSN 0x5061c2
 remote address: LID 0000 QPN 0x0349 PSN 0x4262d1
 remote address: LID 0000 QPN 0x034a PSN 0x7f42e7
 remote address: LID 0000 QPN 0x034b PSN 0xb087a5
 remote address: LID 0000 QPN 0x034c PSN 0x4a9e90
---------------------------------------------------------------------------------------
 #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
 65536 20000 111.72 111.70 0.213057
---------------------------------------------------------------------------------------

```

and then:

```auto
me@gx10-00:~$ mpirun -np 2 -H 10.0.0.50:1,10.0.0.60:1 --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH $HOME/nccl-tests/build/all_gather_perf
Invalid MIT-MAGIC-COOKIE-1 key
Invalid MIT-MAGIC-COOKIE-1 key
Warning: Permanently added '10.0.0.60' (ED25519) to the list of known hosts.
Invalid MIT-MAGIC-COOKIE-1 key
# nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
# Rank 0 Group 0 Pid 49848 on gx10-00 device 0 [000f:01:00] NVIDIA GB10
# Rank 1 Group 0 Pid 62485 on gx10-01 device 0 [000f:01:00] NVIDIA GB10
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)        
    33554432 4194304 float none -1 1106.13 30.33 15.17 0 2315.56 14.49 7.25 0
# Out of bounds values : 0 OK
# Avg bus bandwidth : 11.2064 
#
# Collective test concluded: all_gather_perf
#

```

So, the only remaining question is - are those GB/s numbers correct for 2 GX10’s connected with a single cable?

---

<div class="post-metadata">

**Author:** ![mashie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mashie/32/503632_2.png) [@mashie](https://forums.developer.nvidia.com/u/mashie)\
**Post date:** [August 30, 2026, 9:14pm UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/14 "2026-08-30T21:14:22Z")

</div>

After you installed DACs in the CX7 you need to power off and unplug the GB10’s for a minute as the NIC needs a full power drain for the discovered settings to set the very first time.

---

<div class="post-metadata">

**Author:** ![hiroshiya](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hiroshiya/32/528526_2.png) [@hiroshiya](https://forums.developer.nvidia.com/u/hiroshiya)\
**Post date:** [August 31, 2026, 12:16am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/15 "2026-08-31T00:16:04Z")

</div>

200 Gb/s is 25 GB/s so your `busbw`s are a bit off. Maybe it’s because you’re running a very small test - try `path/to/all_gather_perf -b 16G -e 16G -f 2` instead. For `busbw`, you should get something around 23~24 GB/s.

---

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 31, 2026, 3:27pm UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/16 "2026-08-31T15:27:21Z")

</div>

> After you installed DACs in the CX7 you need to power off and unplug the GB10’s for a minute as the NIC needs a full power drain for the discovered settings to set the very first time.

Interesting. I didn’t seem to have problems with this - but I rebooted so many times and they had been connected for 2 days at that point so maybe it wasn’t something I ran into.

---

<div class="post-metadata">

**Author:** ![mashie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mashie/32/503632_2.png) [@mashie](https://forums.developer.nvidia.com/u/mashie)\
**Post date:** [August 31, 2026, 3:36pm UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/17 "2026-08-31T15:36:17Z")

</div>

It doesn’t happen to everyone but it is very common, as in a post or two a week on these forums.

---

<div class="post-metadata">

**Author:** ![mattropolis](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mattropolis](https://forums.developer.nvidia.com/u/mattropolis)\
**Post date:** [August 31, 2026, 3:37pm UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/18 "2026-08-31T15:37:17Z")

</div>

> [@hiroshiya](#):
>
> path/to/all\_gather\_perf -b 16G -e 16G -f 2

Here’s what I get when running that command:

> ```auto
> $ mpirun -np 2 -H 10.0.0.50:1,10.0.0.60:1 --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
> Warning: Permanently added '10.0.0.60' (ED25519) to the list of known hosts.
> # nccl-tests version 2.20.0 (b4d5bee-dirty) nccl-headers=23007 nccl-library=23007
> # Collective test starting: all_gather_perf
> # nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
> #
> # Using devices
> # Rank 0 Group 0 Pid 1363361 on gx10-00 device 0 [000f:01:00] NVIDIA GB10
> # Rank 1 Group 0 Pid 1353266 on gx10-01 device 0 [000f:01:00] NVIDIA GB10
> #
> # out-of-place in-place
> # size count type redop root time algbw busbw #wrong time algbw busbw #wrong
> # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)        
> 17179869184 2147483648 float none -1 519884 33.05 16.52 0 480074 35.79 17.89 0
> # Out of bounds values : 0 OK
> # Avg bus bandwidth : 17.2079 
> #
> # Collective test concluded: all_gather_perf
> #
> 
> ```

Looks like i’m still below par. Is this a setting issue or possibly a bad cable/port?

I was able to get eugr/spark-vllm-docker up and going with a wide variety of the major models (deepseek v4, qwen, etc). They seem to all work.

---

<div class="post-metadata">

**Author:** ![hiroshiya](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hiroshiya/32/528526_2.png) [@hiroshiya](https://forums.developer.nvidia.com/u/hiroshiya)\
**Post date:** [September 1, 2026, 4:17am UTC](https://forums.developer.nvidia.com/t/2x-gb10-nccl-tests-wont-build/381746/19 "2026-09-01T04:17:59Z")

</div>

If your inference throughput is as expected, then I’d say you are good to go. Debugging those tests does get annoying, and if real-world usage isn’t affected, then I won’t bother fixing them.

Your test results do look weird, though. ~17 GB/s doesn’t really make sense, because it’s much lower than 25 GB/s but also much higher than 12.5 GB/s. I think a lower-grade QSFP cable (QSFP28) goes up to 100 Gbps, so it’s hard to tell where ~17 GB/s comes from.

Your `ib_write_bw` result shows around 111 Gbps on a single port, which is also weird. I think each logic port is capped at 100 Gbps, and my single-port `BW average` is around 99 Gbps.

Still, if you are happy with your inference throughput, then those numbers probably don’t really matter.