I have 2 Mellanox cards (MCX415A-CCAT & MCX416A-CCAT)connected back to back using a Mellanox cable (MCP1600-C002E30N).
The 2 servers are Dell T130 with the latest Centos installed & upgraded. I installed the latest Mellanox drivers & firmware.
I’m using iperf & iperf3 to push the cards close to 100g but I’m getting around 30g
Any help would be appreciated.
When I check with mlnx_tune -r I get this:
ConnectX-4 Device Status on PCI 01:00.0
FW version 12.25.1020
Warning: PCI Width x8 >>> PCI width status is below PCI capabilities. Check PCI configuration in BIOS.
OK: PCI Speed 8GT/s
PCI Max Payload Size 256
PCI Max Read Request 512
Local CPUs list [0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 7]
The slot is confirmed pcie3 X16
I’m answering my own question. It turned out that the server I’m using does not support PCIe with 16 lanes width, only 8 lanes width. The PCIe slot was labeled X16 but that turned out to mean 16 connectors, but still only 8 lanes.
You can always refer to the produce brief to make sure that the adapter supports 16 lanes
Both MCX415A-CCAT & MCX416A-CCAT supports x16
- MCX415A-CCAT ConnectX-4 EN network interface card, 100GbE single-port QSFP28, PCIe3.0 x16, tall bracket, ROHS R6 14.2cm x 6.9cm (Low Profile)
- MCX416A-CCAT ConnectX-4 EN network interface card, 100GbE dual-port QSFP28, PCIe3.0 x16, tall bracket, ROHS R6
In this case the issue is related to the server that is limited only for x8 .