My ConnectX-6 DX takes anywhere from 2 to 15 mins to negotiate the link on FS S5860-20SQ switch and completely fails to establish the link on Ubiquiti USW Enterprise XG 24 switch. Once the link is established, the card works fine with no reliability or performance issues.
The card has the latest FW and was installed on Gigabyte E251-U70 server and then moved it to a PC to rule out host issues. I also tried to use disable Auto negotiation and configure link mode force (–link_mode_force with mlxlink) and set the same duplex, speed, FEC parameters on both sides of the connections, it didn’t help much. Link establishment in Force mode is a bit faster but still takes minutes.
We also have ConnectX-4 LX card that works fine with either of the switches and link establishments takes a few seconds.
It would be great to know what’s different in link negotiation context between 6DX and 4LX and what can be done about it.
Driver Version : 2.80.25134.0
Firmware Version : 22.32.1010
Port Number : 1
Bus Type : PCI-E 8.0 GT/s x4
Link Speed : 25.0 Gbps/Full Duplex
Part Number : MCX621102AC-ADAT
Serial Number : MT2144X07371
Device Id : 4125
Revision Id : 0
Current MAC Address : 08-C0-EB-8B-DC-58
Permanent MAC Address : 08-C0-EB-8B-DC-58
Network Status : Connected
Adapter Friendly Name : Ethernet 6
Port Type : ETH
IPv4 Address #1 : 10.0.50.99
IPv6 Address #1 : fe80::3964:ec49:84c4:1718%15
Cable I tried (The same cables work perfectly for ConnectX-4LX card):
Mellanox MCP2M00-A00AE30N (25GbE DAC)
Mellanox MFA2P10-A005 (25GbE AOC)