In previous cases exhibiting this error/syndrome, we found that the issue stemmed from the use of a kernel prior to version 5.12 that did not include support for the adapter’s real-time clock functionality.
In kernels prior to 5.12, the kernel will try to create QPs with free running timestamp, which is not supported when REAL_TIME_CLOCK_ENABLE is set, thus causing the syndrome exhibited here.
We would recommend either:
Ensure you are using a kernel version that supports this feature as well as ensuring that your applications are attempting to use the ts_format=RT option when opening QPs
or -
Disabling it on the adapter via mlxconfig from the Mellanox Firmware Tools package and utilizing the Free Running Timestamp functionality in your software environment instead.
With MFT installed, start the MST service and display your devices’ MST names:
# mst start
# mst status -v
Once you have the device name, you can set the REAL_TIME_CLOCK_ENABLE option to disabled with the following command:
# mlxconfig -d /dev/mst/ set REAL_TIME_CLOCK_ENABLE=0
For downloads and more information on the MFT package, please review the following links:
If you require further assistance with debugging, we would recommend opening a Support case regarding the issue. If you do not have a current Support contract, please reach out to the team at Networking-contracts@nvidia.com and they can assist you with obtaining one.