ASUS Ascent GX10 — Second ConnectX-7 PCIe slot wired at Gen5 x2 instead of x4 (50% bandwidth loss vs DGX Spark FE)
Summary
The ASUS Ascent GX10 is marketed as functionally equivalent to the NVIDIA DGX
Spark FE, with ASUS’ own FAQ stating the only differentiator is “thermal
design”. My measurements on a 3-node mixed cluster (2× NVIDIA DGX Spark FE +
1× ASUS GX10) show that the GX10’s second ConnectX-7 PCIe endpoint
permanently negotiates at Gen5 x2 instead of x4, halving the available
bandwidth on that NIC. This is a hardware-level asymmetry between the two
products, not a software issue, with no AER errors and a confirmed root-cause
in the upstream root complex.
Affected Product
- Model: ASUSTeK GX10
- System Version: 5.36_GX10DGX
- BIOS: AMI
GX10DGX.0103.2026.0129.1152, dated 2026-01-29 - OS: DGX OS (Ubuntu 24.04, ARM64)
- NIC: 2× NVIDIA/Mellanox ConnectX-7 (MT2910), exposed as 4 PCIe functions
(one QSFP56-DD physical port presents as 2 PCI functions)
Reference: NVIDIA DGX Spark FE
On both DGX Spark FE units in the cluster, all four ConnectX-7 functions
negotiate the expected Gen5 x4:
0000:01:00.0 LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x4
0000:01:00.1 LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x4
0002:01:00.0 LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x4
0002:01:00.1 LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x4
iperf3 over a direct DAC QSFP56-DD link, 8 streams, MTU 9000:
~110 Gbps per lane (consistent with PCIe Gen5 x4 ~126 Gbps theoretical).
Observed on ASUS GX10
0000:01:00.0 LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x4
0000:01:00.1 LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x4
0002:01:00.0 LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x2 (downgraded)
0002:01:00.1 LnkCap: Speed 32GT/s, Width x4 LnkSta: Speed 32GT/s, Width x2 (downgraded)
The card itself is rated x4 (LnkCap), but the upstream port at 0002:00:00.0
only exposes x2 lanes physically — the downgrade is therefore on the GX10
mainboard / SoC side, not the NIC side.
iperf3 across the same direct DAC QSFP56-DD link, same client/server pair as
above, only difference being the GX10 endpoint is on bus 0002:
53 Gbps per lane — exactly half.
Kernel evidence (dmesg)
[0.079518] pci 0002:01:00.0: 63.014 Gb/s available PCIe bandwidth,
limited by 32.0 GT/s PCIe x2 link at 0002:00:00.0
(capable of 126.028 Gb/s with 32.0 GT/s PCIe x4 link)
[3.413380] mlx5_core 0002:01:00.0: 63.014 Gb/s available PCIe bandwidth,
limited by 32.0 GT/s PCIe x2 link at 0002:00:00.0
(capable of 126.028 Gb/s with 32.0 GT/s PCIe x4 link)
The kernel itself flags the bandwidth limitation at boot, on every boot,
deterministically.
PCIe AER (Advanced Error Reporting)
All AER counters (correctable, fatal, non-fatal) on 0002:01:00.0 and
0002:01:00.1 read zero — the link is stable at x2, this is not a
fallback due to signal integrity issues, this is the negotiated steady state.
TOTAL_ERR_COR 0
TOTAL_ERR_FATAL 0
TOTAL_ERR_NONFATAL 0
A pci device reset and a full cold reboot both leave the negotiation at x2.
Throughput summary (3-node cluster)
| Link pair | Bus on each side | Aggregate over 2 lanes |
|---|---|---|
| DGX Spark ↔ DGX Spark | x4 / x4 | ~220 Gbps |
| DGX Spark ↔ GX10 | x4 / x2 | ~163 Gbps |
| GX10 ↔ DGX Spark | x4 / x2 | ~163 Gbps |
A 3-node mesh with this asymmetry means any collective operation (NCCL
all-reduce, vLLM tensor-parallel, distributed training) will see a 25%
performance drop on any pair involving the GX10’s second NIC.
Reproduction
# 1. Read PCIe link status on both bus 0000 and 0002
for slot in $(lspci | grep -i mellanox | cut -d' ' -f1); do
echo "=== $slot ==="
lspci -s "$slot" -vv | grep -E "LnkCap:|LnkSta:"
done
# 2. Confirm via dmesg
dmesg | grep -E "0002:01:00|mlx5_core 0002"
# 3. Confirm AER counters are zero (rules out signal degradation)
cat /sys/bus/pci/devices/0002:01:00.0/aer_dev_correctable
cat /sys/bus/pci/devices/0002:01:00.0/aer_dev_fatal
# 4. Throughput test (peer must be running iperf3 -s on the matching IP)
iperf3 -c <peer_ip_on_bus_0002_subnet> -B <local_ip> -p 5201 -t 10 -P 8
Request
-
Please confirm whether the x2 wiring of the second ConnectX-7 endpoint is
intentional (mainboard design choice) or a manufacturing variance. -
If intentional, please document this in the GX10 datasheet and Tech Specs
page. The current ASUS FAQ and product page do not mention any difference
in PCIe topology vs the DGX Spark FE — only “thermal design”. -
If a future BIOS revision can remap PCIe lanes on this slot to expose x4,
please confirm the timeline. -
Customers buying GX10 to mix with DGX Spark FE in a multi-node cluster
(which the FAQ explicitly endorses, “stacking of 3 devices is supported”)
need to know this asymmetry exists upfront, as it affects every NCCL/RoCE
workload they will run.
Comparable threads on NVIDIA Developer Forums
Other DGX Spark FE users have reported PCIe topology issues, see:
- NCCL bandwidth capped at 3 GB/s, GPU PCIe topology reports Gen1 x1 on DGX Spark FE
- Dual DGX Spark: NCCL capped at 2.80 GB/s + ib_write_bw crashes at 128KB syndrom 0x88 — matches thread 366266 with additional RoCE degradation
The issue reported here is different (deterministic Gen5 x2 vs x4, GX10
specific) but in the same family of “PCIe topology not matching the marketed
spec”.