[CX4]A question of PCIe Link enable/disable via ITP

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

We populated Connectx-4 card on the PCIe slot and performed Link enable/disable via ITP command as below:
“pcie.lt_loop(socket, port, testType=2, loopCount=N”

The following error messages were shown in Linux demsg:

mlx5_core 0000:0b:00.0: poll_health:770: Fatal error 1 detected

mlx5_core 0000:0b:00.1: poll_health:770: Fatal error 1 detected
mlx5_core 0000:0b:00.1: mlx5_error_sw_reset:231: start
mlx5_core 0000:0b:00.0: mlx5_error_sw_reset:231: start

Was the result as expected or a issue?
Does the codes “poll_health:770” and “Fatal error 1” mean?

2

Hi,

What you’re seeing is expected. When ITP toggles the PCIe link via pcie.lt_loop, the device becomes unreachable for a moment while the mlx5_core driver is still bound and polling it. The driver notices and prints those messages it doesn’t mean the card is bad.

Quick decode:

  • poll_health - the driver’s periodic health check on the device.
  • Fatal error 1 -sensor index 1 = MLX5_SENSOR_PCI_COMM_ERR, i.e. config/BAR reads returned all-1s because the link was down.
  • mlx5_error_sw_reset: start -the driver’s automatic recovery kicking in.

To run the link-toggle test cleanly, unbind the driver first so it doesn’t poll the device during the test:

echo 0000:0b:00.0 > /sys/bus/pci/drivers/mlx5_core/unbind
echo 0000:0b:00.1 > /sys/bus/pci/drivers/mlx5_core/unbind

Run pcie.lt_loop, then rescan when you’re done:

echo 1 > /sys/bus/pci/rescan

The fatal-error messages should go away.