DGX Spark 200GbE direct QSFP link negotiates 200G but payload is ~12 Gbps

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Summary

I have two NVIDIA DGX Spark systems connected directly with one QSFP/CX7 cable, following the NVIDIA “Connect Two Sparks” / “Spark Stacking” guidance. The link reports 200000Mb/s on both systems, but TCP and RDMA bandwidth tests are consistently limited to about 12.2-12.7 Gbps in both directions.

NVIDIA documentation referenced:

The playbook says:

  • One QSFP cable is sufficient for direct 200GbE between two DGX Spark systems.
  • Use an interface that shows Up.
  • Disregard interfaces starting with enP2p... and use only enp1....
  • Full bandwidth can be achieved with one QSFP cable.

Physical Setup

  • Two DGX Spark systems.
  • One QSFP/CX7 cable.
  • Cable connected between the same rear ConnectX-7 port on both units.
  • Pull tab/ring tab oriented toward the top of the DGX Spark.
  • Both systems see link up on enp1s0f0np0.

Systems

Node 1

Hostname: spark-c5d5

PRETTY_NAME="Ubuntu 24.04.4 LTS"
Linux spark-c5d5 6.17.0-1021-nvidia #21-Ubuntu SMP PREEMPT_DYNAMIC Wed May 27 19:14:05 UTC 2026 aarch64
NVIDIA driver: 580.159.03
GPU: NVIDIA GB10

Node 2

Hostname: spark-8ac1

PRETTY_NAME="Ubuntu 24.04.4 LTS"
Linux spark-8ac1 6.17.0-1021-nvidia #21-Ubuntu SMP PREEMPT_DYNAMIC Wed May 27 19:14:05 UTC 2026 aarch64
NVIDIA driver: 580.159.03
GPU: NVIDIA GB10

Network Configuration

Configured according to the one-cable playbook, using only enp1s0f0np0:

spark-c5d5: enp1s0f0np0 192.168.100.10/24
spark-8ac1: enp1s0f0np0 192.168.100.11/24

The duplicate enP2p... interface has no IP assigned.

Node 1 ibdev2netdev

roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)

Node 2 ibdev2netdev

roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)

Link State

Both nodes report 200G link on enp1s0f0np0:

Speed: 200000Mb/s
Duplex: Full
Auto-negotiation: on
Link detected: yes

Driver and firmware on both nodes:

driver: mlx5_core
version: 6.17.0-1021-nvidia
firmware-version: 28.45.4028 (NVD0000000087)
bus-info: 0000:01:00.0

PCIe link on both nodes:

current_link_speed: 32.0 GT/s PCIe
current_link_width: 4
max_link_speed: 32.0 GT/s PCIe
max_link_width: 4

Connectivity

Ping succeeds both ways over the QSFP network:

spark-c5d5 -> 192.168.100.11: 0% packet loss, ~1 ms
spark-8ac1 -> 192.168.100.10: 0% packet loss, ~1 ms

Bandwidth Tests

TCP / iperf3

Command used:

# On receiver
iperf3 -s -1 -B 192.168.100.11

# On sender
iperf3 -c 192.168.100.11 -P 8 -t 15

Result:

spark-c5d5 -> spark-8ac1: 12.2 Gbits/sec

Reverse direction:

# On receiver
iperf3 -s -1 -B 192.168.100.10

# On sender
iperf3 -c 192.168.100.10 -P 8 -t 15

Result:

spark-8ac1 -> spark-c5d5: 12.4 Gbits/sec

RDMA / perftest

Commands used previously:

# Receiver
ib_write_bw -d rocep1s0f0 -x 0 -F --report_gbits -s 8388608 -D 10

# Sender
ib_write_bw -d rocep1s0f0 -x 0 -F --report_gbits -s 8388608 -D 10 <remote-ip>

Observed results were consistently around:

12.7 Gbits/sec

This was seen with both ib_write_bw and ib_send_bw.

Things Already Tried

  • Followed the NVIDIA playbook guidance to use only the enp1... interface and disregard enP2p....
  • Removed IP assignment from enP2p1s0f0np0.
  • Tested both directions using iperf3.
  • Tested RDMA with ib_write_bw and ib_send_bw.
  • Verified both nodes have matching kernel, driver, and firmware.
  • Verified PCIe is negotiated at max capability.
  • Verified link reports 200000Mb/s.
  • Tried toggling Ethernet pause off with ethtool -A ... rx off tx off; no improvement.
  • Reseated/reoriented cable. Link comes up, but throughput remains ~12 Gbps.

Question

What is the expected iperf3 and NCCL/IB bandwidth for two DGX Spark systems connected directly with one approved QSFP/CX7 cable?

Given that the link reports 200G but both TCP and RDMA payload tests are limited to ~12 Gbps, is there an additional DGX Spark-specific firmware, driver, cable, power, or ConnectX-7 configuration step required to enable full 200GbE bandwidth?

2

Power off, unplug power for 1min, plug back in, power on, enjoy 200Gb/s

You are the third one in 24h asking the same thing.

I don’t know if the first time a ConnectX-7 port has something connected it is reconfiguring something. However for changes to go active on those cards you need to drain the power.

So yes, this is why pretty much every new user appears to run into the exactly same issue, myself included. It never happens a second time.

3

Thanks Mashie — that fixed it.

We shut both DGX Sparks down, unplugged power from both units for about 60 seconds, then powered them back up. After the full power drain, the QSFP/ConnectX path recovered:

iperf3 on 192.168.200.x: about 111 Gbits/sec
iperf3 on 192.168.201.x: about 111 Gbits/sec
Reverse direction: about 111 Gbits/sec
ib_write_bw on rocep1s0f0: about 109 Gbits/sec

So the issue was exactly as you described: the ConnectX-7 side needed a full power drain, not just a reboot. Appreciate the quick pointer.

4

Same was for me. Always needs reconnect