Slow or failing link negotiation on ConnectX-6 DX SFP28 card

My ConnectX-6 DX takes anywhere from 2 to 15 mins to negotiate the link on FS S5860-20SQ switch and completely fails to establish the link on Ubiquiti USW Enterprise XG 24 switch. Once the link is established, the card works fine with no reliability or performance issues.

The card has the latest FW and was installed on Gigabyte E251-U70 server and then moved it to a PC to rule out host issues. I also tried to use disable Auto negotiation and configure link mode force (–link_mode_force with mlxlink) and set the same duplex, speed, FEC parameters on both sides of the connections, it didn’t help much. Link establishment in Force mode is a bit faster but still takes minutes.

We also have ConnectX-4 LX card that works fine with either of the switches and link establishments takes a few seconds.

It would be great to know what’s different in link negotiation context between 6DX and 4LX and what can be done about it.

Driver Version : 2.80.25134.0

Firmware Version : 22.32.1010

Port Number : 1

Bus Type : PCI-E 8.0 GT/s x4

Link Speed : 25.0 Gbps/Full Duplex

Part Number : MCX621102AC-ADAT

Serial Number : MT2144X07371

Device Id : 4125

Revision Id : 0

Current MAC Address : 08-C0-EB-8B-DC-58

Permanent MAC Address : 08-C0-EB-8B-DC-58

Network Status : Connected

Adapter Friendly Name : Ethernet 6

Port Type : ETH

IPv4 Address #1 : 10.0.50.99

IPv6 Address #1 : fe80::3964:ec49:84c4:1718%15

Cable I tried (The same cables work perfectly for ConnectX-4LX card):

Mellanox MCP2M00-A00AE30N (25GbE DAC)

Mellanox MFA2P10-A005 (25GbE AOC)

Hello,

Unfortunately, neither of the switches that you are attempting to link with the adapter appear within the Firmware Compatible Products listings of the ConnectX-4 LX or the ConnectX-6 DX’s Release Notes. As such, they have not been validated for use with these adapters. Please contact the switch vendors for support.

For a complete listing of tested and validated switches, cables, and modules, please see the following links:

Firmware Compatible Products section of the ConnectX-6 DX Firmware Release Notes:

https://docs.nvidia.com/networking/display/ConnectX6DxFirmwarev22321010/Firmware+Compatible+Products

Firmware Compatible Products section of the ConnectX-4 LX Firmware Release Notes:

https://docs.nvidia.com/networking/display/ConnectX4LxFirmwarev14321010/Firmware+Compatible+Products

Thank you,

-Nvidia Network Support