Hi, I’m using IBV_EXP_WR_CQE_WAIT in Cross Channel feature to support cross-QP synchronization. However, I found that using this verb could degrade the performance of one-sided read/write.
For example, I use multiple QPs at server side, one of which is connected to the client (i.e., QP A). I found that posting IBV_EXP_WR_CQE_WAIT verb in other QPs at server side could prolong the latency of one-sided read/write request sent by the client through QP A. The latency of READ can increase by more than 10%.
Why is this happening? Is this performance degradation a hardware implementation issue?
The hardware that I’m using is MT4115 ConnectX-4 card.
The performance degradation you observe when using IBV_EXP_WR_CQE_WAIT in Cross Channel mode is a result of the way the ConnectX-4 NIC manages cross-QP synchronization internally. This is a hardware/firmware implementation detail and is expected behavior given the feature’s complexity and resource requirements. To mitigate the impact, use this verb judiciously and consider alternative synchronization mechanisms if low latency for unrelated QPs is a priority.
If there are any additional questions or issues, please feel free to open a case with enterprisesupport@nvidia.com, and it will be handled based on entitlement.