Dual DGX Spark Network issue

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, , ,
1

Hello,

Can anyone help me out.

I just got my second dgx spark and im trying to figure out networking. I asked chatgps and i manged to set it up but im not getting more thabt 55Gbit/s.

Using iperf to test speed and i see errors on lanes.

I’ve run multiple tests. DGX Spark 1 → DGX Spark 2 (both 1-1 and 2-2 ports), DGX Spark 1 → DGX Spark 1, DGX Spark 2 → DGX Spark 2
I get max ~ 55,5 Gbits/sec using iperf.

admin@gx1:~$ sudo ethtool -S enp1s0f1np1 | grep -E 'rx_err_lane_[0-3]_phy|rx_crc_errors_phy|rx_symbol_err_phy|rx_pcs_symbol_err_phy'

     rx_crc_errors_phy: 0

     rx_symbol_err_phy: 0

     rx_pcs_symbol_err_phy: 0

     rx_err_lane_0_phy: 33742

     rx_err_lane_1_phy: 12988

     rx_err_lane_2_phy: 0

     rx_err_lane_3_phy: 0

admin@gx2:~$ sudo ethtool -S enp1s0f1np1 | grep -E 'rx_err_lane_[0-3]_phy|rx_crc_errors_phy|rx_symbol_err_phy|rx_pcs_symbol_err_phy'

     rx_crc_errors_phy: 0

     rx_symbol_err_phy: 0

     rx_pcs_symbol_err_phy: 0

     rx_err_lane_0_phy: 22886

     rx_err_lane_1_phy: 109193

     rx_err_lane_2_phy: 0

     rx_err_lane_3_phy: 0

admin@gx1:~$ sudo ethtool enp1s0f1np1 | egrep 'Speed|Duplex|Link detected'

        Speed: 200000Mb/s

Duplex: Full

Link detected: yes

admin@gx2:~$ sudo ethtool enp1s0f1np1 | egrep 'Speed|Duplex|Link detected'

        Speed: 200000Mb/s

Duplex: Full

Link detected: yes

Is his a bad cable ?

I’m trying to figure out if i want to keep the second spark or not. If i cant test models…

Thanks

2

Hello,

First, only connect port/cage 1 to port/cage 1 and 2 → 2, don’t mix the connections.
You won’t get the correct numbers with iPerf, use mpirun like NCCL Test Bandwidth is only 3GB/s between 2 DGX Spark using QSFP cable - #10 by sajid0405103

If you want a quick setup of your cluster network, use https://sparkrun.dev, it’s our community tool for cluster management. There’s a section to configure the connectX-7 network through the wizard:

Or you can see the networking guide:

3 
# Using devices
#  Rank  0 Group  0 Pid  23864 on  gx10-xxx1 device  0 [000f:01:00] NVIDIA GB10
#  Rank  1 Group  0 Pid   9826 on  gx10-xxx2 device  0 [000f:01:00] NVIDIA GB10
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong 
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)             (us)  (GB/s)  (GB/s)         
  1073741824     268435456     float     sum      -1  84996.2   12.63   12.63       0  84811.9   12.66   12.66       0
  2147483648     536870912     float     sum      -1   168078   12.78   12.78       0   168146   12.77   12.77       0
  4294967296    1073741824     float     sum      -1   334706   12.83   12.83       0   333810   12.87   12.87       0
  8589934592    2147483648     float     sum      -1   666428   12.89   12.89       0   665854   12.90   12.90       0
 17179869184    4294967296     float     sum      -1  1330192   12.92   12.92       0  1329798   12.92   12.92       0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 12.8165 
#
# Collective test concluded: all_reduce_perf

after firmware update and everything, still getting only half speed and errors are till there in ethtool lanes

i also tried with one path vs both and getting 13 on one path and 6 on another and 12 on both

after setting up everything i tried iperf3 again, getting 110 on one lane and 55 on the second, errors keep growing after each test.
only conclusion is slot/cable problem (probably cable)

4

What kind of cable you have and how long is it? Plays a major role.

5

The cable is:

“0.5m (1.6ft) NVIDIA/Mellanox NJAAKKR-0006/NJAAKK-0006/LMTQF022-SD-R Compatible 400G DAC QSFP112 4x100G PAM4 to QSFP112 4x100G PAM4 Passive Direct Attach Copper Cable”

but i’ve managed to get the problem down to the spark

0000:01:00.0:
LnkCap: Speed 32GT/s, Width x4
LnkSta: Speed 32GT/s, Width x2 (downgraded)

0000:01:00.1:
LnkCap: Speed 32GT/s, Width x4
LnkSta: Speed 32GT/s, Width x2 (downgraded)

0002:01:00.0:
LnkCap: Speed 32GT/s, Width x4
LnkSta: Speed 32GT/s, Width x4

0002:01:00.1:
LnkCap: Speed 32GT/s, Width x4
LnkSta: Speed 32GT/s, Width x4

always getting one lane downgraded to x2

mlx5_core 0000:01:00.0: 63.014 Gb/s available PCIe bandwidth, limited by 32.0 GT/s PCIe x2 link
capable of 126.028 Gb/s with 32.0 GT/s PCIe x4 link

latest firmware

FW Version: 28.45.4028
FW Release Date: 10.6.2025
Product Version: 28.45.4028
PSID: NVD0000000087

@0rand @raphael.amorim

6

Good stuff! My own from FS.COM is doing amazing today. 22.5 GB/s and 50-60% decode boost on Nemo 3 Super. I was hoping for 30% :)

7

If you have PCIe degradation. Please try performing an AC power cycle by shutting off your Spark, disconnecting the power cable, waiting a few minutes, and then plug back in.