Hello.
NVIDIA/Mellanox ConnectX-6 Dx (MT4125)
- Ubuntu 24.04
- Linux kernel: 6.17
- mlx5 drivers: in-kernel driver
After a complete AC power removal and a subsequent cold boot, both ConnectX-6 Dx adapters report:
[ 11.407954] mlx5_core 0000:ab:00.0: mlx5_pcie_event:322:(pid 12): PCIe slot power capability was not advertised.
[ 12.014922] mlx5_core 0000:ab:00.1: mlx5_pcie_event:322:(pid 386): PCIe slot power capability was not advertised.
[ 12.597107] mlx5_core 0000:bd:00.0: mlx5_pcie_event:322:(pid 12): PCIe slot power capability was not advertised.
[ 13.216092] mlx5_core 0000:bd:00.1: mlx5_pcie_event:322:(pid 755): PCIe slot power capability was not advertised.
However, the standard PCIe Slot Power Limit information is present and correct:
-
the Root Port advertises 75 W in Slot Capabilities;
-
Auto Slot Power Limit Disableis clear; -
the ConnectX PCIe Device Capabilities contain Captured Slot Power Limit = 75 W;
-
the ConnectX MPEIN register contains
pci_power = 75; -
at the same time MPEIN reports
pwr_status = 0.
A normal host reboot does not fix the state.
A level-3 firmware/PCI reset of one HCA changes the behavior of that HCA only. Immediately after the reset the driver can still report that slot power was not advertised, but approximately five seconds later both PFs of the reset HCA report:
PCIe slot advertised sufficient power (75W).
The second physical HCA, which was not reset, remains in the original “not advertised” state.
After a subsequent normal host reboot, the HCA that received the level-3 reset still reports sufficient power (75 W), while the untouched HCA still reports power capability was not advertised.
After completely removing AC power from the server and powering it on again, the problem returns.
Relevant state and Mellanox utility output
Cold boot, before firmware reset:
Root Port Slot Power Limit = 75 W
Auto Slot Power Limit Disable = 0
Endpoint Captured Slot Power = 75 W
Mellanox devices:
# mst status -v
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA VFIO
ConnectX6DX(rev:0) /dev/mst/mt4125_pciconf1.1 bd:00.1 rocep189s0f1 net-ens29f1np1 1
ConnectX6DX(rev:0) /dev/mst/mt4125_pciconf1 bd:00.0 rocep189s0f0 net-ens29f0np0 1
ConnectX6DX(rev:0) /dev/mst/mt4125_pciconf0.1 ab:00.1 rocep171s0f1 net-ens28f1np1 1
ConnectX6DX(rev:0) /dev/mst/mt4125_pciconf0 ab:00.0 rocep171s0f0 net-ens28f0np0 1
Root port:
# lspci -ts 0000:ab:00.0
-[0000:aa]---01.0-[ab]----00.0
# lspci -s 0000:aa:01.0 -vv |grep -i powerlimit
Slot #28, PowerLimit 75W; Interlock- NoCompl-
# lspci -ts 0000:bd:00.0
-[0000:bc]---01.0-[bd]----00.0
# lspci -s 0000:bc:01.0 -vv |grep -i powerlimit
Slot #29, PowerLimit 75W; Interlock- NoCompl-
MPEIN on ab:00.0:
# mlxreg -d /dev/mst/mt4125_pciconf0 --reg_name MPEIN --get --indexes "node=0,pcie_index=0,depth=0"
Sending access register...
Field Name | Data
========================================
node | 0x00000000
pcie_index | 0x00000000
depth | 0x00000000
DPNv | 0x00000000
link_speed_enabled | 0x00000010
link_width_enabled | 0x00000010
link_speed_active | 0x00000010
link_width_active | 0x00000010
lane0_physical_position | 0x0000000f
num_of_vfs | 0x00000000
num_of_pfs | 0x00000002
bdf0 | 0x0000ab00
lane_reversal | 0x00000001
port_type | 0x00000000
pwr_status | 0x00000000
max_payload_size | 0x00000002
max_read_request_size | 0x00000005
pci_power | 0x0000004b
link_peer_max_speed | 0x00000010
flit_sup | 0x00000000
precode_sup | 0x00000000
flit_active | 0x00000000
precode_active | 0x00000000
device_status | 0x00000009
========================================
MPEIN on bd:00.0:
# mlxreg -d /dev/mst/mt4125_pciconf1 --reg_name MPEIN --get -indexes "node=0,pcie_index=0,depth=0"
Sending access register...
Field Name | Data
========================================
node | 0x00000000
pcie_index | 0x00000000
depth | 0x00000000
DPNv | 0x00000000
link_speed_enabled | 0x00000010
link_width_enabled | 0x00000010
link_speed_active | 0x00000010
link_width_active | 0x00000010
lane0_physical_position | 0x0000000f
num_of_vfs | 0x00000000
num_of_pfs | 0x00000002
bdf0 | 0x0000bd00
lane_reversal | 0x00000001
port_type | 0x00000000
pwr_status | 0x00000000
max_payload_size | 0x00000002
max_read_request_size | 0x00000005
pci_power | 0x0000004b
link_peer_max_speed | 0x00000010
flit_sup | 0x00000000
precode_sup | 0x00000000
flit_active | 0x00000000
precode_active | 0x00000000
device_status | 0x00000009
========================================
The important inconsistency is:
pci_power = 0x4b = 75 W
pwr_status = 0
So the numeric slot power is already visible through the HCA firmware interface while the firmware status still indicates that slot power is not advertised/readable.
Reproduction
1. Cold AC power-on
Completely remove AC power from the server, restore power, and boot Ubuntu.
Representative dmesg:
[ 10.988703] mlx5_core 0000:ab:00.0: firmware version: 22.47.3576
[ 10.988727] mlx5_core 0000:ab:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 11.364374] mlx5_core 0000:ab:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 11.371628] mlx5_core 0000:ab:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 11.380753] mlx5_core 0000:ab:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 11.398933] mlx5_core 0000:ab:00.0: Port module event: module 0, Cable unplugged
[ 11.407954] mlx5_core 0000:ab:00.0: mlx5_pcie_event:322:(pid 12): PCIe slot power capability was not advertised.
[ 11.596923] mlx5_core 0000:ab:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 11.608270] mlx5_core 0000:ab:00.1: firmware version: 22.47.3576
[ 11.613187] mlx5_core 0000:ab:00.1: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 11.984581] mlx5_core 0000:ab:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 11.991987] mlx5_core 0000:ab:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 11.999982] mlx5_core 0000:ab:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 12.012578] mlx5_core 0000:ab:00.1: Port module event: module 1, Cable unplugged
[ 12.014922] mlx5_core 0000:ab:00.1: mlx5_pcie_event:322:(pid 386): PCIe slot power capability was not advertised.
[ 12.166786] mlx5_core 0000:ab:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 12.176970] mlx5_core 0000:bd:00.0: firmware version: 22.47.3576
[ 12.178029] mlx5_core 0000:bd:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 12.555686] mlx5_core 0000:bd:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 12.561428] mlx5_core 0000:bd:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 12.563811] mlx5_core 0000:bd:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 12.580046] mlx5_core 0000:bd:00.0: Port module event: module 0, Cable unplugged
[ 12.597107] mlx5_core 0000:bd:00.0: mlx5_pcie_event:322:(pid 12): PCIe slot power capability was not advertised.
[ 12.794059] mlx5_core 0000:bd:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 12.806134] mlx5_core 0000:bd:00.1: firmware version: 22.47.3576
[ 12.808787] mlx5_core 0000:bd:00.1: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 13.191672] mlx5_core 0000:bd:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 13.199045] mlx5_core 0000:bd:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 13.208209] mlx5_core 0000:bd:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 13.212416] mlx5_core 0000:bd:00.1: Port module event: module 1, Cable unplugged
[ 13.216092] mlx5_core 0000:bd:00.1: mlx5_pcie_event:322:(pid 755): PCIe slot power capability was not advertised.
[ 13.391668] mlx5_core 0000:bd:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 13.404063] mlx5_core 0000:ab:00.1 ens28f1np1: renamed from eth3
[ 13.409467] mlx5_core 0000:ab:00.0 ens28f0np0: renamed from eth0
[ 13.428252] mlx5_core 0000:bd:00.1 ens29f1np1: renamed from eth6
[ 13.434033] mlx5_core 0000:bd:00.0 ens29f0np0: renamed from eth5
[ 18.191153] mlx5_core 0000:ab:00.0 ens28f0np0: Link down
[ 18.932065] mlx5_core 0000:ab:00.1 ens28f1np1: Link down
[ 19.763742] mlx5_core 0000:bd:00.0 ens29f0np0: Link down
[ 20.460972] mlx5_core 0000:bd:00.1 ens29f1np1: Link down
A normal host reboot reproduces the same condition.
2. Confirm the Mellanox firmware state
Run:
# mlxreg -d /dev/mst/mt4125_pciconf0 --reg_name MPEIN --get --indexes "node=0,pcie_index=0,depth=0"
# mlxreg -d /dev/mst/mt4125_pciconf1 --reg_name MPEIN --get --indexes "node=0,pcie_index=0,depth=0"
Both HCAs report:
pwr_status | 0x00000000
pci_power | 0x0000004b
3. Reset only one physical HCA
Example:
# mlxfwreset -d /dev/mst/mt4125_pciconf1 reset -y
The reset level for device, /dev/mst/mt4125_pciconf1 is:
3: Driver restart and PCI reset
Continue with reset?[y/N] y
Reset level: 3, Reset type: 0, Reset sync: 0, Reset method: 0
-I- Sending Reset Command To Fw -Done
-I- Stopping Driver -Done
-I- Resetting PCI -Done
-I- Starting Driver -Done
-I- Restarting MST -Done
-I- FW was loaded successfully.
# mlxreg -d /dev/mst/mt4125_pciconf1 --reg_name MPEIN --get --indexes "node=0,pcie_index=0,depth=0" |grep -i pwr
pwr_status | 0x00000001
# dmesg |grep -i mlx5
[ 256.470600] mlx5_core 0000:bd:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[ 256.486702] mlx5_core 0000:bd:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[ 259.462841] mlx5_core 0000:bd:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[ 260.375178] mlx5_core 0000:bd:00.0: E-Switch: cleanup
[ 260.681917] mlx5_core 0000:bd:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[ 260.702733] mlx5_core 0000:bd:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[ 263.395823] mlx5_core 0000:bd:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[ 264.315253] mlx5_core 0000:bd:00.1: E-Switch: cleanup
[ 266.196945] mlx5_core 0000:bd:00.0: enabling device (0100 -> 0102)
[ 266.197222] mlx5_core 0000:bd:00.0: firmware version: 22.47.3576
[ 266.197248] mlx5_core 0000:bd:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 266.579391] mlx5_core 0000:bd:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 266.579623] mlx5_core 0000:bd:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 266.581771] mlx5_core 0000:bd:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 266.586397] mlx5_core 0000:bd:00.0: mlx5_pcie_event:322:(pid 386): PCIe slot power capability was not advertised.
[ 266.776769] mlx5_core 0000:bd:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 266.788749] mlx5_core 0000:bd:00.0 ens29f0np0: renamed from eth0
[ 266.802511] mlx5_core 0000:bd:00.1: enabling device (0100 -> 0102)
[ 266.803080] mlx5_core 0000:bd:00.1: firmware version: 22.47.3576
[ 266.803140] mlx5_core 0000:bd:00.1: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 267.306094] mlx5_core 0000:bd:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 267.306254] mlx5_core 0000:bd:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 267.307225] mlx5_core 0000:bd:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 267.311469] mlx5_core 0000:bd:00.1: mlx5_pcie_event:322:(pid 868): PCIe slot power capability was not advertised.
[ 267.504377] mlx5_core 0000:bd:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 267.507475] mlx5_core 0000:bd:00.1 ens29f1np1: renamed from eth0
[ 272.295557] mlx5_core 0000:bd:00.0: Port module event: module 0, Cable unplugged
[ 272.295597] mlx5_core 0000:bd:00.1: Port module event: module 1, Cable unplugged
[ 272.296394] mlx5_core 0000:bd:00.1: mlx5_pcie_event:331:(pid 386): PCIe slot advertised sufficient power (75W).
[ 272.296443] mlx5_core 0000:bd:00.0: mlx5_pcie_event:331:(pid 12): PCIe slot advertised sufficient power (75W).
Immediately after the reset the same HCA can initially report:
[ 266.586397] mlx5_core 0000:bd:00.0: mlx5_pcie_event:322:(pid 386): PCIe slot power capability was not advertised.
[ 267.311469] mlx5_core 0000:bd:00.1: mlx5_pcie_event:322:(pid 868): PCIe slot power capability was not advertised.
Approximately five seconds later it changes to:
[ 272.296394] mlx5_core 0000:bd:00.1: mlx5_pcie_event:331:(pid 386): PCIe slot advertised sufficient power (75W).
[ 272.296443] mlx5_core 0000:bd:00.0: mlx5_pcie_event:331:(pid 12): PCIe slot advertised sufficient power (75W).
4. Reboot normally
[ 11.226985] mlx5_core 0000:ab:00.0: firmware version: 22.47.3576
[ 11.227009] mlx5_core 0000:ab:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 11.592897] mlx5_core 0000:ab:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 11.600205] mlx5_core 0000:ab:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 11.609496] mlx5_core 0000:ab:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 11.613648] mlx5_core 0000:ab:00.0: Port module event: module 0, Cable unplugged
[ 11.627831] mlx5_core 0000:ab:00.0: mlx5_pcie_event:322:(pid 386): PCIe slot power capability was not advertised.
[ 11.810812] mlx5_core 0000:ab:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 11.830625] mlx5_core 0000:ab:00.1: firmware version: 22.47.3576
[ 11.832940] mlx5_core 0000:ab:00.1: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 12.195289] mlx5_core 0000:ab:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 12.202551] mlx5_core 0000:ab:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 12.210667] mlx5_core 0000:ab:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 12.217979] mlx5_core 0000:ab:00.1: Port module event: module 1, Cable unplugged
[ 12.224437] mlx5_core 0000:ab:00.1: mlx5_pcie_event:322:(pid 12): PCIe slot power capability was not advertised.
[ 12.398112] mlx5_core 0000:ab:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 12.408050] mlx5_core 0000:bd:00.0: firmware version: 22.47.3576
[ 12.413001] mlx5_core 0000:bd:00.0: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 12.780060] mlx5_core 0000:bd:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 12.787229] mlx5_core 0000:bd:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 12.788431] mlx5_core 0000:bd:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 12.793751] mlx5_core 0000:bd:00.0: Port module event: module 0, Cable unplugged
[ 12.794575] mlx5_core 0000:bd:00.0: mlx5_pcie_event:331:(pid 774): PCIe slot advertised sufficient power (75W).
[ 12.970721] mlx5_core 0000:bd:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 12.982090] mlx5_core 0000:bd:00.1: firmware version: 22.47.3576
[ 12.987002] mlx5_core 0000:bd:00.1: 252.048 Gb/s available PCIe bandwidth (16.0 GT/s PCIe x16 link)
[ 13.366561] mlx5_core 0000:bd:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[ 13.367074] mlx5_core 0000:bd:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 13.368640] mlx5_core 0000:bd:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 13.374510] mlx5_core 0000:bd:00.1: Port module event: module 1, Cable unplugged
[ 13.375735] mlx5_core 0000:bd:00.1: mlx5_pcie_event:331:(pid 386): PCIe slot advertised sufficient power (75W).
[ 13.548696] mlx5_core 0000:bd:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 13.556670] mlx5_core 0000:ab:00.0 ens28f0np0: renamed from eth0
[ 13.595603] mlx5_core 0000:ab:00.1 ens28f1np1: renamed from eth3
[ 13.601090] mlx5_core 0000:bd:00.1 ens29f1np1: renamed from eth6
[ 13.607164] mlx5_core 0000:bd:00.0 ens29f0np0: renamed from eth4
[ 18.492527] mlx5_core 0000:ab:00.0 ens28f0np0: Link down
[ 19.195037] mlx5_core 0000:ab:00.1 ens28f1np1: Link down
[ 19.902787] mlx5_core 0000:bd:00.0 ens29f0np0: Link down
[ 20.727591] mlx5_core 0000:bd:00.1 ens29f1np1: Link down
After the next normal reboot, the HCA that received the level-3 reset immediately reports:
[ 12.794575] mlx5_core 0000:bd:00.0: mlx5_pcie_event:331:(pid 774): PCIe slot advertised sufficient power (75W).
[ 13.375735] mlx5_core 0000:bd:00.1: mlx5_pcie_event:331:(pid 386): PCIe slot advertised sufficient power (75W).
The HCA that was not reset still reports:
[ 11.627831] mlx5_core 0000:ab:00.0: mlx5_pcie_event:322:(pid 386): PCIe slot power capability was not advertised.
[ 12.224437] mlx5_core 0000:ab:00.1: mlx5_pcie_event:322:(pid 12): PCIe slot power capability was not advertised.
5. Remove AC power again
Completely remove AC power from the server and power it on again.
The problem is reproduced and the HCA returns to:
PCIe slot power capability was not advertised.
This shows that the corrected state survives a normal host reboot but is lost after a complete loss of power.
Questions:
- Is this a known ConnectX-6 Dx firmware issue?
- What exactly causes MPEIN
pwr_status=0whenpci_power=75and the PCIe Captured Slot Power Limit is already 75 W?