we met an issue "CREATE_QP(0x500) op_mod(0x0) failed" when using MLX5 poll mode driver

by using linux command dmesg, get information, can you help to the root cause, thanks.

[ 4.734223] mlx5_core 0000:d8:00.0: mlx5_cmd_check:769:(pid 853): CREATE_QP(0x500) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0xd61c0b)

[ 4.734265] infiniband mlx5_0: mlx5_ib_gsi_create_qp:189:(pid 853): unable to create hardware GSI QP. error -22

[ 4.735481] infiniband mlx5_0: Couldn’t create ib_mad QP1

[ 4.736840] infiniband mlx5_0: Couldn’t open port 1

[ 4.738455] mlx5_core 0000:d8:00.0: mlx5_cmd_check:769:(pid 853): CREATE_QP(0x500) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0xd61c0b)

[ 4.739721] infiniband mlx5_0: Port 1 not found

[ 4.739738] infiniband mlx5_0: Couldn’t close port 1 for agents

[ 4.739754] infiniband mlx5_0: Port 1 not found

[ 4.739766] infiniband mlx5_0: Couldn’t close port 1

Hello,

In previous cases exhibiting this error/syndrome, we found that the issue stemmed from the use of a kernel prior to version 5.12 that did not include support for the adapter’s real-time clock functionality.

In kernels prior to 5.12, the kernel will try to create QPs with free running timestamp, which is not supported when REAL_TIME_CLOCK_ENABLE is set, thus causing the syndrome exhibited here.

We would recommend either:

  1. Ensure you are using a kernel version that supports this feature as well as ensuring that your applications are attempting to use the ts_format=RT option when opening QPs
  • or -
  1. Disabling it on the adapter via mlxconfig from the Mellanox Firmware Tools package and utilizing the Free Running Timestamp functionality in your software environment instead.

With MFT installed, start the MST service and display your devices’ MST names:

# mst start

# mst status -v

Once you have the device name, you can set the REAL_TIME_CLOCK_ENABLE option to disabled with the following command:

# mlxconfig -d /dev/mst/ set REAL_TIME_CLOCK_ENABLE=0

For downloads and more information on the MFT package, please review the following links:

MFT 4.18 User Manual:

https://docs.mellanox.com/pages/viewpage.action?pageId=64302942

MFT Downloads:

https://www.mellanox.com/products/adapter-software/firmware-tools

If you require further assistance with debugging, we would recommend opening a Support case regarding the issue. If you do not have a current Support contract, please reach out to the team at Networking-contracts@nvidia.com and they can assist you with obtaining one.

Thank you,

Nvidia Network Support