NCCL single-cable test caps at 100Gbps

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi,

I’m connecting two Sparks with a 200GbE QSFP56 cable according to the playbooks. Connect Two Sparks works and I can see both ports are Up, but I’m having trouble with NCCL for Two Sparks.

After setting the environment variables and running the command below:
mpirun -np 2 -H 169.254.6.140:1,169.254.221.162:1 --mca plm_rsh_agent “ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no” -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH -x UCX_NET_DEVICES=enp1s0f0np0 -x NCCL_SOCKET_IFNAME=enp1s0f0np0 -x NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 -x NCCL_IB_DISABLE=0 -x NCCL_DEBUG=INFO $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2

The best avg. bus bandwidth I get is 15.9, but I thought I should be getting 20+. So far I’ve tried
-applying higher/lower MTU in the netplan (setting both to 9000 instead of 1500 makes the bandwidth worse)

  • adding NCCL_IB_GID_INDEX=3 (which fails bc there are two indexes at most)
  • adding NCCL_IB_QPS_PER_CONNECTION=8

I also tested using ib_write_bw, same issue where the highest speed is 100Gbps.

My ibdev2netdev seems fine:

ibdev2netdev
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

It’s like the other port to allow 200Gbps through one cable isn’t working properly or something. How could I fix this?

Thank you!

3

You only need one cable to have 200Gbps speed. Running the playbook with a second cable will limit your bandwidth to 100Gbps

4

Current firmware has a performance regression in ConnectX 7 workloads. We are all seeing a drop from ~24 GB/s to ~16 GB/s on NCCL tests. NVIDIA is aware and is working on a fix.

5

Yup, I’m using just one cable.

6

I see, thanks so much! Would that apply for ib_write_bw as well though?

7

I believe so. I can test on my sparks later today.

8

Fixed with the February 2026 system update.

9

I don’t believe it is fixed. I am running two Sparks (Asus Ascent) on the latest firmware:

# nccl-tests version 2.18.2 nccl-headers=22809 nccl-library=22809
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0 unalign: 0
#
# Using devices
#  Rank  0 Group  0 Pid 252495 on      hades device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid 117267 on hephaestus device  0 [000f:01:00] NVIDIA GB10
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)
    33554432       4194304     float    none      -1  1087.39   30.86   15.43       0  1057.44   31.73   15.87       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 15.6474
#
# Collective test concluded: all_gather_perf
#
10

It’s fixed, but only on Founders Edition so far. They said OEM will release the update on their own schedule.

11

Hi @eugr could you please share your numbers so we can see a baseline.

12

Sure, here we go:

run-nccl-tests.sh:

#!/bin/bash

# Set environment variables
export CUDA_HOME="/usr/local/cuda"
export MPI_HOME="/usr/lib/aarch64-linux-gnu/openmpi"
export NCCL_HOME="$HOME/nccl/build/"
export LD_LIBRARY_PATH="$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH"

export UCX_NET_DEVICES=enp1s0f1np1
export NCCL_SOCKET_IFNAME=enp1s0f1np1
export OMPI_MCA_btl_tcp_if_include=enp1s0f1np1
export NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1
export NCCL_IB_DISABLE=0

# Run the all_gather performance test across both nodes
mpirun -np 2 -H 192.168.177.11:1,192.168.177.12:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2

# nccl-tests version 2.17.9 nccl-headers=22903 nccl-library=22903
# Collective test starting: all_gather_perf
# nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 1 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   3907 on      spark device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid   4084 on     spark2 device  0 [000f:01:00] NVIDIA GB10
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)
 17179869184    2147483648     float    none      -1   355390   48.34   24.17       0   353734   48.57   24.28       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 24.227
#
# Collective test concluded: all_gather_perf
#
13

Thanks! so your all-gather perf is 24g/s

14

Yep. It was 16 GB/s before this update (and 24 GB/s before February update that caused performance degradation).

15

Thanks. 24 is still pretty low. Which cable are you using?

16

24 GB/s is not low, it’s almost 200 Gb/s - the top advertised speed for the interface.

17

I’m using Amphenol NJAAKR-0006, QSFP112 cable.