Intermittent timeout when modifying QP to RTR with RoCE

[ctang1207]

I am using CX6 cards (4123).
I create 4 QPs, wiring to 4 different remote QPs, the first two QP were successful. The third one timeout (errno=110).
At first I think there is a network issue that the NIC can’t resolve the remote IP’s MAC. so I print the MAC entry before and after the ibv_modify_qp(RTR) call.
before, there is no MAC entry, but right after the call, the MAC entry exists. However, ibv_modify_qp(RTR) already failed.
This is intermittent and hard to reproduce.
In the Mellanox driver, what is the reason to generate errno=110 with ibv_modify_qp(RTR) ?

Thanks for any suggestion for solve this problem.

By the way, two years ago, there was a similar question:

Intermittent Connection timed out while setting RC QP to RTR

Hi All, I am using Connectx5-EX on 5.19.0-45-generic. One side creates an RC qp and sends it’s qp info over TCP and the other side upon receiving this message creates it’s own RC QP and sets it to RTR followed by setting to RTS, and it responds back over same TCP connection, to the initiator which then sets it’s own QP to RTR then RTS. This works most of the times but sometimes I get connection time out from ibv_modify_qp() while setting to RTR. I have two dual function Connectx5-Ex so total fo…

But it was closed and I did not get the answer from the dialog.