ConnectX-7 Inter-Spark Link Capped at ~13 Gbps (Expected 200 Gbps) — PCIe Power Throttling (27W)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Hardware: 2x NVIDIA DGX Spark (GB10 SoC) Firmware: ConnectX-7, v28.45.4028 (current) Cable: Amphenol QSFP cable (link up at 200 Gbps / 2 lanes) Power: Official NVIDIA DGX Spark power adapters OS: Ubuntu (stock DGX Spark image, fully updated)

Hi all,

I have two DGX Sparks connected via the QSFP high-speed link. Both interfaces negotiate at 200 Gbps (confirmed via ethtool), but actual throughput is capped at approximately 13 Gbps — whether tested via TCP (iperf3) or RDMA (ib_write_bw). NVIDIA’s own Performance Benchmarking Guide shows expected RDMA bandwidth of 92–97 Gbps per interface (~190 Gbps combined).

What I’ve Tried

  1. Verified traffic is on CX-7, not WiFi. Initially traffic was routing over WiFi instead of the CX-7 interfaces. After assigning static IPs via netplan to the CX-7 interfaces, traffic correctly flows over the high-speed link.

  2. iperf3 testing — single stream, 16 parallel streams (-P 16), and 4 separate iperf3 instances on different ports. All cap at ~12.5–13 Gbps total.

  3. RDMA testing — ib_write_bw using the exact commands from NVIDIA’s benchmarking guide:

   # Server:
   ib_write_bw -d rocep1s0f0 -F --report_gbits -q 4 -D 30

   # Client:
   ib_write_bw -d rocep1s0f0 -F --report_gbits -q 4 -D 30 <server-ip>

Result: 13.42 Gbps (expected: 92+ Gbps per interface, ~190 Gbps combined)

  1. Ring buffers increased from 1024 to 8192 — no improvement.

  2. Qdisc changed from fq_codel to mq — no improvement.

  3. MTU set to 9978 (jumbo frames) on all interfaces — no improvement.

  4. CPU utilization checked during test: 98.7% idle — not a CPU bottleneck.

  5. Rate limiting checked via mlnx_qos — all rates unlimited, no throttling configured.

  6. Firmware confirmed current: v28.45.4028 on both Sparks.

  7. SWIOTLB/bounce buffers — not present, ruled out.

Root Cause Found: PCIe Power Throttling

Both Sparks show this in dmesg:

mlx5_core 0000:01:00.0: mlx5_pcie_event: Detected insufficient power on the PCIe slot (27W).
mlx5_core 0002:01:00.0: mlx5_pcie_event: Detected insufficient power on the PCIe slot (27W).

This appears on both ports of both Sparks (4 messages total). The ConnectX-7 NIC is reporting that it’s not receiving enough power from the PCIe slot, which is almost certainly causing it to throttle performance.

Both Sparks are using the official NVIDIA DGX Spark power adapters — no third-party power supplies.

PCIe Topology

The ConnectX-7 operates in multi-host mode with two PCIe Gen5 x4 root ports:

  • Domain 0000: enp1s0f0np0 / rocep1s0f0

  • Domain 0002: enP2p1s0f0np0 / roceP2p1s0f0

lspci shows: Speed 32GT/s (ok), Width x4 (ok) — PCIe Gen5 x4 is correct.

Questions

  1. Is the 27W PCIe power warning a known issue on DGX Spark?

  2. Is there a firmware or BIOS/UEFI update that addresses this power allocation?

  3. Has anyone else achieved the expected 92+ Gbps throughput on the inter-Spark link?

  4. Could this be a hardware issue requiring RMA?

Any guidance would be appreciated. Happy to provide additional diagnostic output.

System details:

  • Both Sparks fully updated (apt update && apt upgrade)

  • Firmware: ConnectX-7 v28.45.4028

2

The insufficient power message caused a lot of head scratching for the DGX Spark users! Looks like the firmware is lying to the network driver:

From lspci output:

	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W

The SlotPowerLimit is 0W.

As you already noticed the link looks healthy, i.e. Speed 32GT/s. However, the SlotPowerLimit reading might cause the network driver to throttle, thinking there’s not enough power for full bandwidth.

3

I had the same thing. connected my two sparks, ran the test and got 13 Gb/s

I applied updates and restarted both and re-ran and got 111 Gb/s.

So either recent updates fixed it, or they just needed a reboot

4

I can share another DGX Spark data point that looks related.

Setup:

  • 4x DGX Spark
  • Ubuntu 24.04
  • Kernel: 6.17.0-1021-nvidia
  • NVIDIA driver: 580.159.03
  • mlx5 firmware: 28.45.4028
  • Direct QSFP connections, no switch
  • Link reports 200000Mb/s and PCIe reports 32GT/s x4
  • dmesg includes mlx5_pcie_event messages such as:
    Detected insufficient power on the PCIe slot (27W)
  • PCIe SlotPowerLimit appears as 0W

Initial symptom:

  • Some RDMA directions were capped around 12.8-13.5Gb/s.
  • Other directions on the same physical setup were around 93-105Gb/s.
  • The low-speed direction was reproduced with both ib_write_bw and a CUDA pinned-host RDMA test using cudaHostAlloc + ibv_reg_mr.
  • Moving ports, reversing cable orientation, and swapping cable assemblies did not explain the severe 13Gb/s cap.

What recovered it:

  • Rebooting the affected endpoint after the final cable topology was connected recovered the severe 13Gb/s state.
  • Example before reboot:
    • ib_write_bw: ~12.86Gb/s in the affected direction
    • CUDA pinned-host RDMA: ~11.8Gb/s
  • Example after reboot:
    • ib_write_bw: ~100-104Gb/s
    • CUDA pinned-host RDMA: ~105-108Gb/s

One important detail:

  • Reconnecting/moving QSFP cables could re-trigger the low-speed state.
  • The practical recovery procedure was:
    1. Connect the final intended topology.
    2. Run a short ib_write_bw test in both directions.
    3. If one endpoint is still around 13Gb/s, reboot that endpoint with the final cabling still connected.
    4. Retest.

Residual observation:

  • After fixing the 13Gb/s state, we still see a stable smaller directional delta:
    • about 105.5Gb/s in one direction
    • about 97.5Gb/s in the other direction
  • This residual delta did not change after:
    • rebooting all endpoints
    • using the alternate RDMA device on the same physical QSFP path
    • setting Ethernet MTU 9000 / RDMA active_mtu 4096
    • testing larger RDMA message sizes
    • temporarily setting PCIe ASPM policy to performance
  • So the severe 13Gb/s cap and the smaller 97/105Gb/s delta appear to be different issues.

Hope this helps. I would be interested to know whether NVIDIA expects a firmware/kernel fix for the 13Gb/s hotplug/low-power state, and whether the remaining ~8% directional delta is expected on DGX Spark direct QSFP links.

5

Do you have those nodes connected in a ring topology? How are you routing the traffic between the nodes that aren’t directly connected?

The ConnectX-7 power warnings in the logs are just cosmetic so not helpful for any troubleshooting.

6

Sorry for the confusion, and thanks for the reply. The current topology is not a 4-node ring. We currently have two direct QSFP pairs: 01-02 and 03-04 on enp1s0f0np0/rocep1s0f0.

During the investigation I temporarily rewired nodes for direct pair tests such as 02-03 and 01-04. Those were direct physical links at the time of each test, not routed traffic through another Spark. We did not test non-direct multi-hop routing between QSFP links.

Good to know the PCIe power warnings are cosmetic. The 13Gb/s state still appeared after QSFP hotplug while ethtool showed 200Gb/s and PCIe remained 32GT/s x4, and rebooting the affected endpoint after final cabling recovered it. After recovery, both direct pairs are in the ~97-105Gb/s range in both directions, with a small stable directional delta remaining.

7

No worries, was just curious if you had built something out of the norm.

When I had my link act up in a similar way it was fixed by power off, unplug the power lead for a minute and try again. It seems to be the fix for most strange behaviours. Although saying that, it has only happened once here.

8

Hi @Ank-Chy

Did updating your system resolve your issue?

10

Additional confirmed DGX Spark data point: rebooting both endpoints with the
final QSFP topology already connected recovered the severe ~13 Gb/s-per-interface
state.

Setup:

  • 2x DGX Spark, direct QSFP connection
  • DGX OS 7.5.0 / Ubuntu 24.04.4
  • Kernel: 6.17.0-1029-nvidia
  • NVIDIA driver: 580.173.02
  • ConnectX-7 firmware: 28.45.4028 (NVD0000000087)
  • Both PCIe paths: 32.0 GT/s x4
  • Ethernet link: 200000 Mb/s, full duplex
  • Ethernet MTU 9000 / RDMA active_mtu 4096

Before reboot, NVIDIA’s two-interface ib_write_bw procedure produced:

  • Interface 1: 13.39 Gb/s
  • Interface 2: 13.76 Gb/s
  • Aggregate: 27.15 Gb/s

Both RDMA interfaces were ACTIVE/LINK_UP. Jumbo ping passed with no packet loss,
and CRC, symbol, discard, and TX error counters remained at zero. The same 27 W
PCIe warning discussed in this thread was present.

Recovery procedure:

  1. Connect the final intended QSFP topology.
  2. Do not move or reconnect the cable.
  3. Reboot both DGX Sparks.
  4. Revalidate both interfaces and run the official test simultaneously.

Post-reboot test configuration:

  • Normal verbs, no RDMA-CM
  • One QP per interface
  • 64 KiB messages
  • -F
  • PCIe relaxed ordering ON
  • 15-second bounded duration
  • Both interfaces tested simultaneously

Results:

  • Interface 1: 98.01 Gb/s
  • Interface 2: 98.01 Gb/s
  • Aggregate: 196.02 Gb/s

This passes NVIDIA’s 184 Gb/s gate and is comparable to the 189.85 Gb/s example
in the official guide. No CRC, symbol, discard, or TX errors appeared during the
test. Only corrected FEC bits increased.

This strongly supports an initialization/hot-plug state rather than a persistent
Netplan, MTU, PCIe-width, cable-error, QoS, or CPU limitation. It does not prove
that the 27 W warning itself caused the throttling.

Could NVIDIA confirm whether this is a known ConnectX-7 firmware/driver
initialization issue after QSFP hot-plug, and whether a permanent fix is planned?

11

Yes, the issue is resolved and the issue can be marked as closed from my end