ConnectX-5 Ex (MCX556A-EDAT): PCIe Gen4 equalization completes but speed change to 16GT/s never asserted — link stays at 8GT/s

Infrastructure & Networking Base Command Manager
1

Firmware: 16.35.4030 and 16.35.4554 LTS (both affected)
PSID: MT_0000000009
Part: MCX556A-EDAT (ConnectX-5 Ex VPI, dual-port QSFP28, PCIe4.0 x16)
Platform: AMD Ryzen AI Max+ 395 “Strix Halo”, Framework Desktop mainboard
Host PCIe controller: AMD Strix/Strix Halo GPP Bridge [1022:150b]
OS: Fedora Linux, kernel 6.x
Cards tested: 4 cards total — 3x Vogzone MCX556A-EDAT, 1x LR-Link MCX556A-EDAT

Summary

ConnectX-5 Ex cards on this platform complete PCIe Gen4 equalization successfully across all four phases, but the link speed remains at 8GT/s (Gen3). The speed change to 16GT/s (Gen4) is never asserted after equalization. All PCIe capability registers on both the host root port and the NIC endpoint correctly advertise Gen4, equalization completes without error, but the link never transitions to 16GT/s.

An Intel E810-CQDA2 card in the same physical slot on the same platform trains to PCIe Gen4 x4 (16GT/s) without issue, confirming the platform is capable of Gen4 and the issue is specific to the ConnectX-5 firmware.

Environment

Item Detail
NIC MCX556A-EDAT (ConnectX-5 Ex VPI)
PSID MT_0000000009
Firmware tested 16.35.4030, 16.35.4554 LTS
Platform AMD Ryzen AI Max+ 395 (Strix Halo)
Mainboard Framework Desktop (Ryzen AI Max)
Host root port AMD GPP Bridge [1022:150b]
Slot wiring PCIe x16 physical, x4 electrical
Reference card Intel E810-CQDA2 (trains Gen4 x4 correctly on same slot)
Cards tested 4 units from 2 vendors, all identical behaviour

Diagnostic Data

Host root port (00:03.2) PCIe capability registers

LnkCap:   Port #0, Speed 16GT/s, Width x4      ← host advertises Gen4
LnkCap2:  Supported Link Speeds: 2.5-16GT/s    ← full speed vector
LnkCtl2:  Target Link Speed: 16GT/s            ← target explicitly set to Gen4
LnkSta:   Speed 8GT/s, Width x4                ← actual negotiated speed

Phy16Sta: EquComplete+ EquPhase1+ EquPhase2+ EquPhase3+ LinkEquRequest-

Gen4 equalization completed successfully on all four phases.

NIC endpoint (c3:00.0) PCIe capability registers

LnkCap:   Port #0, Speed 16GT/s, Width x16     ← NIC advertises Gen4
LnkCap2:  Supported Link Speeds: 2.5-16GT/s    ← full speed vector
LnkCtl2:  Target Link Speed: 8GT/s             ← NIC targets Gen3 (initial state)
LnkSta:   Speed 8GT/s (downgraded), Width x4 (downgraded)

Note the (downgraded) annotation — the NIC itself reports it knows it is running below its capability.

Raw register values (setpci)

# Root port LnkCtl2 (offset 0x88): 0x0043 = Target 8GT/s (Gen3), bit6 set
# NIC endpoint LnkCtl2 (offset 0x90): 0x0003 = Target 8GT/s (Gen3)

# Root port LnkCap (offset 0x64): 0x00737844
#   bits [3:0] = 4 → 16GT/s Gen4
# NIC endpoint LnkCap (offset 0x6c): 0x00417104
#   bits [3:0] = 4 → 16GT/s Gen4

mlxconfig — no NVRAM speed overrides

mlxconfig query on both cards shows all default values. There are no LINK_SPEED_OVERRIDE or equivalent non-default PCIe settings in NVRAM on either card.

Reproduction Steps

  1. Install MCX556A-EDAT in PCIe slot on AMD Strix Halo platform.
  2. Boot system. Card trains at 8GT/s x4 (Gen3) instead of expected 16GT/s x4 (Gen4).
  3. Verify both sides advertise Gen4 capability (confirmed via lspci -vv, setpci reads above).
  4. Manually set both sides to target Gen4 and force retrain with PerformEqualization:
# Set Gen4 target on root port (preserve bit 6)
sudo setpci -s 00:03.2 0x88.w=0x0044

# Set Gen4 target on NIC endpoint  
sudo setpci -s c3:00.0 0x90.w=0x0004

# Force full equalization on retrain
sudo setpci -s 00:03.2 0x274.l=0x00000002

# Trigger retrain link
sudo setpci -s 00:03.2 0x68.w=0x0c60

# After retrain:
sudo lspci -s 00:03.2 -vv | grep -E "LnkSta:|LnkCtl2:|Phy16Sta:"
  1. Result after retrain:
LnkSta:   Speed 8GT/s, Width x4           ← still Gen3
LnkCtl2:  Target Link Speed: 16GT/s       ← target held, not overwritten by BIOS
Phy16Sta: EquComplete+ EquPhase1+ EquPhase2+ EquPhase3+ LinkEquRequest-

Gen4 equalization completes, LnkCtl2 targets are held, but the link speed does not change.

Analysis

The PCIe Gen4 link training sequence has two distinct stages:

  1. Equalization (Phases 1–3): Both sides exchange preset coefficients to optimize the 16GT/s signal. This completes successfully (Phy16Sta: EquComplete+).
  2. Speed change: After equalization, the downstream component (the NIC) is responsible for initiating the transition from the training speed to the negotiated operating speed (16GT/s). This step is not occurring.

All conditions required for the speed change are met:

  • Both LnkCap registers report Gen4 as max speed ✓
  • Both LnkCap2 registers include 16GT/s in the supported speeds vector ✓
  • Both LnkCtl2 registers target 16GT/s ✓
  • Equalization completes without error ✓
  • No NVRAM overrides on the card ✓
  • BIOS does not revert LnkCtl2 targets after bus reset ✓

The Intel E810-CQDA2 on the same slot, same platform, same BIOS trains to 16GT/s x4 without any manual intervention, ruling out a platform or BIOS issue.

The behaviour is consistent across 4 ConnectX-5 Ex cards from 2 different PCB vendors (Vogzone and LR-Link), on 2 separate machines with the same Strix Halo platform, on both firmware 16.35.4030 and 16.35.4554 LTS.

The most likely cause is that the ConnectX-5 firmware’s post-equalization speed change logic does not assert the speed change transition when paired with the AMD GPP Bridge [1022:150b] root port. This could be related to the specific timing, Data Link Feature exchange, or capability advertisement behaviour of this root port during the speed change handshake.

Questions

  1. Is there a known issue or workaround for ConnectX-5 Ex cards not asserting the PCIe speed change to 16GT/s after successful Gen4 equalization on AMD platforms?

  2. Is there an NVRAM configuration parameter (via mlxconfig) that controls the post-equalization speed change behaviour, or that could force the card to initiate the speed change?

  3. Is this being tracked internally? If a fix is in progress, what firmware version is expected to address it?

Additional Notes

  • The AMD Strix Halo platform is a relatively new consumer/prosumer platform (2025). Other users have reported ConnectX-5 cards achieving Gen4 on a Minisforum MS-S1 Max (also Strix Halo), suggesting this may be a marginal or platform-variant-specific interaction.
  • This issue has significant practical impact for anyone building InfiniBand or high-speed RoCE clusters using Strix Halo nodes, as Gen3 x4 yields ~28 Gbps effective bandwidth vs ~56 Gbps at Gen4 x4.