What 10GbE link modes are supported by the MCX4121A-ACAT ConnectX-4 Lx EN NIC (25GbE Dual-Port SFP28) ?

I’ve just installed a MCX4121A-ACAT ConnectX-4 Lx EN NIC in a Linux PC (x86_64, Linux Mint 19.3 (based on Ubuntu 18.04), kernel 5.4.0-72).

The NIC is connected to an external device which supports only 10Gbase-R (e.g. 10Gbase-SR, 10Gbase-CR etc. but not 10Gbase-KR). A link appears to be established, but no data appears to be transmitted or received.

‘ethtool’ reports the following when using a copper cable / transceiver in the 1st SFP28 cage on the MCX4121A-ACAT :

Settings for enp1s0f0:

Supported ports: [ Backplane ]

Supported link modes: 1000baseKX/Full

10000baseKR/Full

25000baseCR/Full

25000baseKR/Full

25000baseSR/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: Yes

Supported FEC modes: None BaseR

Advertised link modes: 10000baseKR/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: No

Advertised FEC modes: None

Speed: 10000Mb/s

Duplex: Full

Port: Direct Attach Copper

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Link detected: yes

Similarly, when using an optical cable / transceiver in the 2nd SFP28 cage :

Settings for enp1s0f1:

Supported ports: [ FIBRE ]

Supported link modes: 1000baseKX/Full

10000baseKR/Full

25000baseCR/Full

25000baseKR/Full

25000baseSR/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: Yes

Supported FEC modes: None BaseR

Advertised link modes: 10000baseKR/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: No

Advertised FEC modes: None

Speed: 10000Mb/s

Duplex: Full

Port: FIBRE

PHYAD: 0

Transceiver: internal

Auto-negotiation: off

Link detected: yes

Thus, it appears that the MCX4121A-ACAT supports base-SR, base-CR and base-KR at 25G, but only base-KR at 10G. Is this correct ?

Further information which may be useful :

  • the MCX4121A-ACAT firmware version is 14.27.1016. Is there any possibility that upgrading to the newest firmware (14.30.1004) is likely to provide support for any other 10G link modes ?
  • output from lspci -v :

01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT

Flags: bus master, fast devsel, latency 0, IRQ 16

Memory at a6000000 (64-bit, prefetchable) [size=32M]

Expansion ROM at a5400000 [disabled] [size=1M]

Capabilities: [60] Express Endpoint, MSI 00

Capabilities: [48] Vital Product Data

Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-

Capabilities: [c0] Vendor Specific Information: Len=18 <?>

Capabilities: [40] Power Management version 3

Capabilities: [100] Advanced Error Reporting

Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

Capabilities: [180] Single Root I/O Virtualization (SR-IOV)

Capabilities: [1c0] #19

Capabilities: [230] Access Control Services

Kernel driver in use: mlx5_core

Kernel modules: mlx5_core

01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT

Flags: bus master, fast devsel, latency 0, IRQ 17

Memory at a8000000 (64-bit, prefetchable) [size=32M]

Expansion ROM at a5300000 [disabled] [size=1M]

Capabilities: [60] Express Endpoint, MSI 00

Capabilities: [48] Vital Product Data

Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-

Capabilities: [c0] Vendor Specific Information: Len=18 <?>

Capabilities: [40] Power Management version 3

Capabilities: [100] Advanced Error Reporting

Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

Capabilities: [180] Single Root I/O Virtualization (SR-IOV)

Capabilities: [230] Access Control Services

Kernel driver in use: mlx5_core

Kernel modules: mlx5_core

Thank you,

Andrew.

Thank you Aleksey for your quick response.

I’ve upgraded the firmware to 14.30.1004, as per dmesg output below :

[ 1.186634] mlx5_core 0000:01:00.0: firmware version: 14.30.1004

[ 1.186664] mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)

[ 1.403056] mlx5_core 0000:01:00.0: E-Switch: Total vports 10, per vport: max uc(1024) max mc(16384)

[ 1.405656] mlx5_core 0000:01:00.0: Port module event: module 0, Cable unplugged

[ 1.417804] mlx5_core 0000:01:00.1: firmware version: 14.30.1004

[ 1.417847] mlx5_core 0000:01:00.1: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)

[ 1.643971] mlx5_core 0000:01:00.1: E-Switch: Total vports 10, per vport: max uc(1024) max mc(16384)

[ 1.647506] mlx5_core 0000:01:00.1: Port module event: module 1, Cable plugged

However, ‘ethtool’ still indicates that only base-KR is supported for 10G. ‘ethtool’ output below is for one of the SFP28 ports with no SFP module or cable connected, so I’d expect that the supported link modes are not affected by any attached module or cable :

Settings for enp1s0f0:

Supported ports: [ FIBRE Backplane ]

Supported link modes: 1000baseKX/Full

10000baseKR/Full

25000baseCR/Full

25000baseKR/Full

25000baseSR/Full

Supported pause frame use: Symmetric

Supports auto-negotiation: Yes

Supported FEC modes: Not reported

Advertised link modes: 1000baseKX/Full

10000baseKR/Full

25000baseCR/Full

25000baseKR/Full

25000baseSR/Full

Advertised pause frame use: Symmetric

Advertised auto-negotiation: Yes

Advertised FEC modes: Not reported

Speed: Unknown!

Duplex: Unknown! (255)

Port: FIBRE

PHYAD: 0

Transceiver: internal

Auto-negotiation: on

Link detected: no

Can you please confirm whether or not the MCX4121A-ACAT supports 10G link modes other than 10GbaseKR (even if using Mellanox validated modules / cables) ?

The modules / cables I am using are not Mellanox ones, but they are a well-known brand, and I don’t think that could be preventing the MCX4121A-ACAT from showing any other supported 10G link modes - particularly in the case shown above with no module / cable connected.

Thanks,

Andrew.

It is always a good idea to use latest firmware. Could be that you need to use supported cables/module. Please, take a look here

https://docs.mellanox.com/display/ConnectX4LxFirmwarev14301004/Firmware+Compatible+Products

A further update to my previous reply : I’ve tried re-connecting and checking the link to the external device mentioned in my first post, and data now appears to be received and transmitted.

‘ethtool’ still does not show any supported 10G link mode other than 10GbaseKR, which I know for certain is not supported by the external device I’m using. However, it seems that the link is now working (using the same optical module / cable as before, so the link must be up using 10GbaseSR). Presumably the firmware update has fixed something,

Thanks again Aleksey for your response.

Andrew.