ConnectX-5: Cannot bring up link after connection state bouncing

Summary

On a ConnectX-5 NIC, if cable is repeatedly connected and disconnected on a frequent basis, the link state will eventually remain down and cannot become up without NIC reset.

System Information

The server tested is running BC-Linux 8.2 with latest MLNX_OFED driver and firmware. Two dual-port ConnectX-5 NICs are installed, each of which has one port connected to switch with a DAC cable. However, only one NIC is necessary to reproduce this issue. The port ens1f0np0 is used here.

Miscellaneous Version Information
[root@localhost ~]# uname -a
Linux localhost.localdomain 4.19.0-240.23.11.el8_2.bclinux.x86_64 #1 SMP Wed Jun 2 16:11:31 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# ofed_info -s
MLNX_OFED_LINUX-5.8-1.0.1.1:
[root@localhost ~]# lspci | grep -i mellanox
3b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
3b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
5e:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
5e:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
[root@localhost ~]# ibv_devinfo
hca_id: mlx5_2
        transport:                      InfiniBand (0)
        fw_ver:                         16.35.1012
        node_guid:                      248a:0703:00a3:e3bc
        sys_image_guid:                 248a:0703:00a3:e3bc
        vendor_id:                      0x02c9
        vendor_part_id:                 4119
        hw_ver:                         0x0
        board_id:                       MT_0000000012
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_3
        transport:                      InfiniBand (0)
        fw_ver:                         16.35.1012
        node_guid:                      248a:0703:00a3:e3bd
        sys_image_guid:                 248a:0703:00a3:e3bc
        vendor_id:                      0x02c9
        vendor_part_id:                 4119
        hw_ver:                         0x0
        board_id:                       MT_0000000012
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_0
        transport:                      InfiniBand (0)
        fw_ver:                         16.35.1012
        node_guid:                      b859:9f03:00c1:f1aa
        sys_image_guid:                 b859:9f03:00c1:f1aa
        vendor_id:                      0x02c9
        vendor_part_id:                 4119
        hw_ver:                         0x0
        board_id:                       MT_0000000012
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: mlx5_1
        transport:                      InfiniBand (0)
        fw_ver:                         16.35.1012
        node_guid:                      b859:9f03:00c1:f1ab
        sys_image_guid:                 b859:9f03:00c1:f1aa
        vendor_id:                      0x02c9
        vendor_part_id:                 4119
        hw_ver:                         0x0
        board_id:                       MT_0000000012
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet
[root@localhost ~]# ethtool ens1f0np0
Settings for ens1f0np0:
        Supported ports: [ Backplane ]
        Supported link modes:   1000baseKX/Full 
                                10000baseKR/Full 
                                40000baseKR4/Full 
                                40000baseCR4/Full 
                                40000baseSR4/Full 
                                40000baseLR4/Full 
                                25000baseCR/Full 
                                25000baseKR/Full 
                                25000baseSR/Full 
                                50000baseCR2/Full 
                                50000baseKR2/Full 
                                100000baseKR4/Full 
                                100000baseSR4/Full 
                                100000baseCR4/Full 
                                100000baseLR4_ER4/Full 
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: None BaseR RS
        Advertised link modes:  1000baseKX/Full 
                                10000baseKR/Full 
                                40000baseKR4/Full 
                                40000baseCR4/Full 
                                40000baseSR4/Full 
                                40000baseLR4/Full 
                                25000baseCR/Full 
                                25000baseKR/Full 
                                25000baseSR/Full 
                                50000baseCR2/Full 
                                50000baseKR2/Full 
                                100000baseKR4/Full 
                                100000baseSR4/Full 
                                100000baseCR4/Full 
                                100000baseLR4_ER4/Full 
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: RS
        Speed: 100000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000004 (4)
                               link
        Link detected: yes

Steps to reproduce

  1. Connect server port to switch. Verify the link is up and running.
  2. Shutdown the switch port. Link state on the server will soon become down.
  3. After 5 seconds, bring up the switch port.
  4. Once server link becomes up, immediately shutdown the switch port again.
  5. Repeat steps 3 and 4 for about 15 times.
  6. Eventually server link will not become up again even if switch port is kept enabled.
[root@localhost ~]# ip link show ens1f0np0
52: ens1f0np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether b8:59:9f:c1:f1:aa brd ff:ff:ff:ff:ff:ff
[root@localhost ~]# mlxlink -d /dev/mst/mt4119_pciconf0

Operational Info
----------------
State                           : Polling
Physical state                  : ETH_AN_FSM_ABILITY_DETECT
Speed                           : N/A
Width                           : N/A
FEC                             : N/A
Loopback Mode                   : No Loopback
Auto Negotiation                : ON

Supported Info
--------------
Enabled Link Speed              : 0xf8f1f0d3 (100G,50G,40G,25G,10G,1G)
Supported Cable Speed           : 0x48101165 (100G,56G,50G,40G,25G,20G,10G,1G)

Troubleshooting Info
--------------------
Status Opcode                   : 2
Group Opcode                    : PHY FW
Recommendation                  : Negotiation failure

Tool Information
----------------
Firmware Version                : 16.35.1012
MFT Version                     : mft 4.22.0-96

If negotiation had been disabled (for both server and switch), the error message would be ‘Other issues’:

[root@localhost ~]# mlxlink -d /dev/mst/mt4119_pciconf0

Operational Info
----------------
State                           : Polling
Physical state                  : ETH_AN_FSM_ABILITY_DETECT
Speed                           : N/A
Width                           : N/A
FEC                             : N/A
Loopback Mode                   : No Loopback
Auto Negotiation                : FORCE - 100G

Supported Info
--------------
Enabled Link Speed              : 0x00f00000 (100G)
Supported Cable Speed           : 0x48101165 (100G,56G,50G,40G,25G,20G,10G,1G)

Troubleshooting Info
--------------------
Status Opcode                   : 36
Group Opcode                    : PHY FW
Recommendation                  : Other issues

Tool Information
----------------
Firmware Version                : 16.35.1012
MFT Version                     : mft 4.22.0-96

Resetting the NIC with mlxfwreset is usually enough to bring link back up. Any ideas on how to resolve this issue? Thanks!

Hello hzx,

Thank you for posting your inquiry to the NVIDIA Developer Forums.

In order to rule out any hardware incompatibilities (which could be responsible for such a behavior), we recommend using a supported cable from the following list:
https://docs.nvidia.com/networking/display/ConnectX5Firmwarev16351012/Firmware+Compatible+Products

Additionally, it’s possible that some non-standard adapter configuration was applied, which could be indirectly responsible for this behavior. We recommend resetting the adapter configuration to default values and retesting, using the ‘mlxconfig’ tool:

mlxconfig -d /dev/mst/mt4119_pciconf0 reset

…And rebooting.

If this behavior is not resolved by the above, we would recommend opening a case with Enterprise Support via the following link for further investigation:
https://enterprise-support.nvidia.com/s/create-case

Best,
NVEX Networking Support