I can't reach even close to 100gb/s speed on a 100gb/s ethernet card.

Infrastructure & Networking Adapters and Cables
1

I have 2 Mellanox cards (MCX415A-CCAT & MCX416A-CCAT)connected back to back using a Mellanox cable (MCP1600-C002E30N).

The 2 servers are Dell T130 with the latest Centos installed & upgraded. I installed the latest Mellanox drivers & firmware.

I’m using iperf & iperf3 to push the cards close to 100g but I’m getting around 30g

Any help would be appreciated.

When I check with mlnx_tune -r I get this:

ConnectX-4 Device Status on PCI 01:00.0

FW version 12.25.1020

Warning: PCI Width x8 >>> PCI width status is below PCI capabilities. Check PCI configuration in BIOS.

OK: PCI Speed 8GT/s

PCI Max Payload Size 256

PCI Max Read Request 512

Local CPUs list [0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 7]

The slot is confirmed pcie3 X16

2

I’m answering my own question. It turned out that the server I’m using does not support PCIe with 16 lanes width, only 8 lanes width. The PCIe slot was labeled X16 but that turned out to mean 16 connectors, but still only 8 lanes.

3

Hi Nass,

You can always refer to the produce brief to make sure that the adapter supports 16 lanes

http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-4_EN_Card.pdf

Both MCX415A-CCAT & MCX416A-CCAT supports x16

  • MCX415A-CCAT ConnectX-4 EN network interface card, 100GbE single-port QSFP28, PCIe3.0 x16, tall bracket, ROHS R6 14.2cm x 6.9cm (Low Profile)
  • MCX416A-CCAT ConnectX-4 EN network interface card, 100GbE dual-port QSFP28, PCIe3.0 x16, tall bracket, ROHS R6

In this case the issue is related to the server that is limited only for x8 .

Thanks,

Samer