NCCL Test Bandwidth is only 3GB/s between 2 DGX Spark using QSFP cable

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
10

I was able to get 24 GB/s with the firmware udpdates. Thanks @s0ne @vgoklani @eugr

ssharlemin@edgexpert-4245:~/dgx-spark-playbooks/nvidia/connect-two-sparks/assets$ mpirun -np 2 -H 192.168.100.10:1,192.168.100.11:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Warning: Permanently added '192.168.100.11' (ED25519) to the list of known hosts.
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

/home/ssharlemin/nccl-tests/build/all_gather_perf: error while loading shared libraries: libnccl.so.2: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
/home/ssharlemin/nccl-tests/build/all_gather_perf: error while loading shared libraries: libnccl.so.2: cannot open shared object file: No such file or directory
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[46420,1],0]
  Exit code:    127
--------------------------------------------------------------------------
ssharlemin@edgexpert-4245:~/dgx-spark-playbooks/nvidia/connect-two-sparks/assets$ export CUDA_HOME="/usr/local/cuda"
export MPI_HOME="/usr/lib/aarch64-linux-gnu/openmpi"
export NCCL_HOME="$HOME/nccl/build/"
export LD_LIBRARY_PATH="$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH"

# Point out-of-band communication to the active Ethernet interfaces
export UCX_NET_DEVICES=enp1s0f0np0,enP2p1s0f0np0
export NCCL_SOCKET_IFNAME=enp1s0f0np0,enP2p1s0f0np0
export OMPI_MCA_btl_tcp_if_include=enp1s0f0np0

# Bind NCCL to the bifurcated 100G logical rails for full 200G multi-rail throughput
export NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
export NCCL_IB_DISABLE=0
ssharlemin@edgexpert-4245:~/dgx-spark-playbooks/nvidia/connect-two-sparks/assets$  mpirun -np 2 -H 192.168.100.10:1,192.168.100.11:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Warning: Permanently added '192.168.100.11' (ED25519) to the list of known hosts.
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

# nccl-tests version 2.18.2 nccl-headers=22809 nccl-library=22809
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
#  Rank  0 Group  0 Pid   5270 on edgexpert-4245 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid   4060 on edgexpert-3a77 device  0 [000f:01:00] NVIDIA GB10
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)
 17179869184    2147483648     float    none      -1   354669   48.44   24.22       0   353648   48.58   24.29       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 24.2545
#
# Collective test concluded: all_gather_perf
#

disabled the channel after the updae though

sudo fwupdmgr disable-remote lvfs-testing
sudo fwupdmgr refresh --force

Thanks for all the response :)

I was able to run some local model ./run-recipe.sh qwen3.5-397b-int4-autoround.yaml --no-ray using GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub @eugr 's docker before the update.

I plan to run QuantTrio/Qwen3-VL-235B-A22B-Instruct-AWQ · Hugging Face in the next hours and see how that goes :)

If i ran into problems I will create a new thread.