# Dual DGX Spark Network issue **URL:** \ **Category:** DGX Spark / GB10\ **Tags:** networking, dgx-spark-issue, spark, dgx\ **Created:** [May 27, 2026, 2:08pm UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532 "2026-05-27T14:08:15Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![voicu.victor.dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@voicu.victor.dev](https://forums.developer.nvidia.com/u/voicu.victor.dev)\ **Post date:** [May 27, 2026, 2:08pm UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532/1 "2026-05-27T14:08:15Z")
Hello, Can anyone help me out. I just got my second dgx spark and im trying to figure out networking. I asked chatgps and i manged to set it up but im not getting more thabt 55Gbit/s. Using iperf to test speed and i see errors on lanes. I’ve run multiple tests. DGX Spark 1 → DGX Spark 2 (both 1-1 and 2-2 ports), DGX Spark 1 → DGX Spark 1, DGX Spark 2 → DGX Spark 2 I get max ~ 55,5 Gbits/sec using iperf. ```auto admin@gx1:~$ sudo ethtool -S enp1s0f1np1 | grep -E 'rx_err_lane_[0-3]_phy|rx_crc_errors_phy|rx_symbol_err_phy|rx_pcs_symbol_err_phy' rx_crc_errors_phy: 0 rx_symbol_err_phy: 0 rx_pcs_symbol_err_phy: 0 rx_err_lane_0_phy: 33742 rx_err_lane_1_phy: 12988 rx_err_lane_2_phy: 0 rx_err_lane_3_phy: 0 admin@gx2:~$ sudo ethtool -S enp1s0f1np1 | grep -E 'rx_err_lane_[0-3]_phy|rx_crc_errors_phy|rx_symbol_err_phy|rx_pcs_symbol_err_phy' rx_crc_errors_phy: 0 rx_symbol_err_phy: 0 rx_pcs_symbol_err_phy: 0 rx_err_lane_0_phy: 22886 rx_err_lane_1_phy: 109193 rx_err_lane_2_phy: 0 rx_err_lane_3_phy: 0 admin@gx1:~$ sudo ethtool enp1s0f1np1 | egrep 'Speed|Duplex|Link detected' Speed: 200000Mb/s Duplex: Full Link detected: yes admin@gx2:~$ sudo ethtool enp1s0f1np1 | egrep 'Speed|Duplex|Link detected' Speed: 200000Mb/s Duplex: Full Link detected: yes ``` Is his a bad cable ? I’m trying to figure out if i want to keep the second spark or not. If i cant test models… Thanks ---
**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\ **Post date:** [May 27, 2026, 2:27pm UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532/2 "2026-05-27T14:27:09Z")
Hello, First, only connect port/cage 1 to port/cage 1 and 2 → 2, don’t mix the connections. You won’t get the correct numbers with iPerf, use mpirun like [NCCL Test Bandwidth is only 3GB/s between 2 DGX Spark using QSFP cable - #10 by sajid0405103](https://forums.developer.nvidia.com/t/nccl-test-bandwidth-is-only-3gb-s-between-2-dgx-spark-using-qsfp-cable/366373/10) If you want a quick setup of your cluster network, use [https://sparkrun.dev](https://sparkrun.dev), it’s our community tool for cluster management. There’s a section to configure the connectX-7 network through the wizard: > **[Setup Wizard Walkthrough](https://sparkrun.dev/getting-started/setup-wizard/)** > > Step-by-step guide through the sparkrun setup wizard for first-time users. Or you can see the networking guide: > **[Networking Best Practices](https://sparkrun.dev/getting-started/networking/)** > > Best practices for configuring DGX Spark networking for multi-node inference clusters. ---
**Author:** ![voicu.victor.dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@voicu.victor.dev](https://forums.developer.nvidia.com/u/voicu.victor.dev)\ **Post date:** [May 27, 2026, 5:54pm UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532/3 "2026-05-27T17:54:02Z")
```auto # Using devices # Rank 0 Group 0 Pid 23864 on gx10-xxx1 device 0 [000f:01:00] NVIDIA GB10 # Rank 1 Group 0 Pid 9826 on gx10-xxx2 device 0 [000f:01:00] NVIDIA GB10 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 1073741824 268435456 float sum -1 84996.2 12.63 12.63 0 84811.9 12.66 12.66 0 2147483648 536870912 float sum -1 168078 12.78 12.78 0 168146 12.77 12.77 0 4294967296 1073741824 float sum -1 334706 12.83 12.83 0 333810 12.87 12.87 0 8589934592 2147483648 float sum -1 666428 12.89 12.89 0 665854 12.90 12.90 0 17179869184 4294967296 float sum -1 1330192 12.92 12.92 0 1329798 12.92 12.92 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 12.8165 # # Collective test concluded: all_reduce_perf ``` after firmware update and everything, still getting only half speed and errors are till there in ethtool lanes i also tried with one path vs both and getting 13 on one path and 6 on another and 12 on both after setting up everything i tried iperf3 again, getting 110 on one lane and 55 on the second, errors keep growing after each test. only conclusion is slot/cable problem (probably cable) ---
**Author:** ![0rand](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/0rand/32/512004_2.png) [@0rand](https://forums.developer.nvidia.com/u/0rand)\ **Post date:** [May 29, 2026, 5:22am UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532/4 "2026-05-29T05:22:34Z")
What kind of cable you have and how long is it? Plays a major role. ---
**Author:** ![voicu.victor.dev](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@voicu.victor.dev](https://forums.developer.nvidia.com/u/voicu.victor.dev)\ **Post date:** [May 30, 2026, 9:10am UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532/5 "2026-05-30T09:10:38Z")
The cable is: “**0.5m (1.6ft) NVIDIA/Mellanox NJAAKKR-0006/NJAAKK-0006/LMTQF022-SD-R Compatible 400G DAC QSFP112 4x100G PAM4 to QSFP112 4x100G PAM4 Passive Direct Attach Copper Cable**” but i’ve managed to get the problem down to the spark 0000:01:00.0: LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x2 (downgraded) 0000:01:00.1: LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x2 (downgraded) 0002:01:00.0: LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x4 0002:01:00.1: LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x4 always getting one lane downgraded to x2 mlx5\_core 0000:01:00.0: 63.014 Gb/s available PCIe bandwidth, limited by 32.0 GT/s PCIe x2 link capable of 126.028 Gb/s with 32.0 GT/s PCIe x4 link latest firmware FW Version: 28.45.4028 FW Release Date: 10.6.2025 Product Version: 28.45.4028 PSID: NVD0000000087 @0rand @raphael.amorim ---
**Author:** ![0rand](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/0rand/32/512004_2.png) [@0rand](https://forums.developer.nvidia.com/u/0rand)\ **Post date:** [May 30, 2026, 8:46pm UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532/6 "2026-05-30T20:46:16Z")
Good stuff! My own from [FS.COM](http://FS.COM) is doing amazing today. 22.5 GB/s and 50-60% decode boost on Nemo 3 Super. I was hoping for 30% :) ---
**Author:** ![aniculescu](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aniculescu/32/382649_2.png) [@aniculescu](https://forums.developer.nvidia.com/u/aniculescu)\ **Post date:** [June 1, 2026, 6:46pm UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532/7 "2026-06-01T18:46:08Z")
If you have PCIe degradation. Please try performing an AC power cycle by shutting off your Spark, disconnecting the power cable, waiting a few minutes, and then plug back in. ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [June 30, 2026, 9:59pm UTC](https://forums.developer.nvidia.com/t/dual-dgx-spark-network-issue/371532/8 "2026-06-30T21:59:41Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.