ConnectX-5 cable error/One or more network ports have been powered down due to insufficient/unadvertised power on the PCIe slot

Infrastructure & Networking Adapters And Cables
#1

I installed a Mellanox MCX516A-CDAT NIC on ARM platform. After plug in a QSFP28 cable (MFA1A00-C003) to both ports for loopback testing, the link can’t be established.

I connected one port of MCX516A-CDAT to the port of the other MCX516A-CDAT on X86 platform, still saw the issues at ARM platform.

Error message below.

mlx5_core 0000:01:00.0: port_module:240:(pid 475): Port module event[error]: module 0, Cable error, One or more network ports have been powered down due to insufficient/unadvertised power on the PCIe slot.

How to fix this issue?

I didn’t see such issues when I installed the MCX416A-CCAT NIC with same QSFP28 cable.

Referring to messages below, shall I set the instructions?

https://community.mellanox.com/s/feed/0D51T00006RVupTSAT

mlxconfig -d /dev/mst/mt4121_pciconf0 set DISABLE_SLOT_POWER_LIMITER=True

I saw the warnings below.

WARNING: YOU ARE ABOUT TO RE-ENABLE ONE OR MORE NETWORK PORTS THAT HAVE BEEN SHUT DOWN DUE TO INSUFFICIENT POWER ON THE PCIE SLOT.

YOUR ACTION MAY RISK PROPER FUNCTIONAL OR PHYSICAL OPERATION OF THE CARD OR OF THE SERVER.

MELLANOX TECHNOLOGIES HEREBY DISCLAIMS ANY AND ALL LIABILITY RELATED TO ANY DAMAGE CAUSED AS A RESULT OF THIS RE-ENABLEMENT.

BY PROCEEDING WITH THIS RE-ENABLEMENT YOU WILL INVALIDATE ANY WARRANTY PROVIDED BY MELLANOX TECHNOLOGIES RELATED TO THIS CARD.

IT IS ADVISED TO CHECK THE CARD’S USER MANUAL FOR POWER SPECIFICATIONS OR TO CONTACT MELLANOX SUPPORT BEFORE PERFORMING THIS ACTION.

#2

Hi Jerry,

Please note that MCX516A-CDAT is ConnectX-5 EX

Kindly refer to this section in the release notes, page 42 “Power MGMT”:

http://www.mellanox.com/pdf/firmware/ConnectX5-FW-16_26_1040-release_notes.pdf

Added support for PCIe Express standard “Slot capability register” message (PCIe base rev 3.1, section 6.9 – “Slot Power Limit Control”) When ConnectX-5 Ex based adapter is inserted to a PCIe slot that supports the reporting of the slot power limit control, the ConnectX-5 Ex may disable the 2nd port if PCIe slot message, showing that the power in this slot is insufficient. If not, both ports will stay enabled. In cases where ConnectX-5 Ex based adapter is inserted to a PCI slot that DOES NOT support the “Slot capability register” message, and the adapters’ configuration is 2 active optic cables/ transceivers, only one port will be enabled (the first inserted optic).

Custom and OEM branded card based on ConnectX-5 Ex may be configured by INI to support/not-support the Power management feature.

In hosts which do not support the “Slot capability register” Message and have enough power to support 2 active optical cable,

the user will have the option to override the configuration resulted from “Slot capability register” by running the following NVconfig command:

• echo “MLNX_RAW_TLV_FILE” > /tmp/power_conf_tlv.cfg; echo “0x00000004 0x00000088 0x00000000 0xc0000000” >> /tmp/power_conf_tlv.cfg

• mlxconfig -d -f /tmp/power_conf_tlv.cfg set_raw

• mlxfwreset -d reset

Thanks,

Samer

#3

Hi, Samer

Great! Thanks a lot for your quick response!

The instructions you shared with me works pretty well!

Thanks,

Jerry