Is there some relation with PCIe port speed/version and mellanox connectx--4 en link detection?

m.serdienis · August 17, 2020, 10:36am

We have 3 servers (2x HP and 1x Intel) with ConnectX-4 Lx EN network interface cards (MCX4121A-ACA_Ax). Two servers (HP) works OK with 25Gbit DACs, but Intel refuses to see link on 25Gbit DACs… The same DACs, the same device at the other end…

If I change DACs to 10Gbit DACs, then the link is detected…

The same firmware on all cards (14.26.1040), tried to update the firmware to the latest (did not helped).

Any idea why 25Gbit DAC do not work? Can it be that the card is faulty?

chenh1 · August 19, 2020, 9:47am

Hi Martins,

When connecting the NIC to the PCIe, it auto-negotiates for the maximum capabilities supported between the network adapter and the CPU. ConnectX-4 Lx EN supports 25Gb/s speed.

Did you try to connect a different NIC to the Intel server in order to exclude a NIC issue?
You can try to force the link to 25 to Auto Negotiation to OFF and force the speed:

mlxlink -d /dev/mst/mt4119_pciconf0 --link_mode_force --speeds

E.G:

mlxlink -d /dev/mst/mt4119_pciconf0 --link_mode_force --speeds 25G

Then you must toggle the link so the above changes will take place:

mlxlink -d mt4119_pciconf0 -a TG

Regards,

Chen

m.serdienis · August 19, 2020, 1:04pm

Sad, but that did not helped :-(

Have not tried other card, because I do not have spare cards, other 2 card are in production servers.

root@SRV3:~# mlxlink -d /dev/mst/mt4117_pciconf0.1

Operational Info

State : Disable

Physical state : ETH_AN_FSM_ENABLE

Speed : N/A

Width : N/A

FEC : N/A

Loopback Mode : N/A

Auto Negotiation : FORCE - 25G

Supported Info

Enabled Link Speed : 0x38000000 (25G)

Supported Cable Speed : 0x38007013 (25G,10G,1G)

Troubleshooting Info

Status Opcode : 1027

Group Opcode : MNG FW

Recommendation : Invalid module memory map. Replace the module/cable.

The same info from different server:

root@SRV1:~# mlxlink -d /dev/mst/mt4117_pciconf0.1

Operational Info

State : Active

Physical state : LinkUp

Speed : 25GbE

Width : 1x

FEC : Standard RS-FEC - RS(528,514)

Loopback Mode : No Loopback

Auto Negotiation : ON

Supported Info

Enabled Link Speed : 0x38007013 (25G,10G,1G)

Supported Cable Speed : 0x38007013 (25G,10G,1G)

Troubleshooting Info

Status Opcode : 0

Group Opcode : N/A

Recommendation : No issue was observed.

Cable information from working server:

root@SRV1:~# mlxcables

Querying Cables …

Cable #1:

Cable name : mt4117_pciconf0.1_cable_1

No FW data to show

-------- Cable EEPROM --------

Identifier : SFP/SFP+/SFP28 (03h)

Technology : Passive Copper Cable, Electrical intra-enclosure

Compliance : Unspecified, 25GBASE-CR CA-N

OUI : 0x006016

Vendor : Amphenol

Serial number : APF9320175966H

Part number : NDCCGJ-E003

Revision : C

Temperature : N/A

Length : 3 m

Cable #2:

Cable name : mt4117_pciconf0_cable_0