why ibv_poll_cq always have about 2 seconds delay before available?

Infrastructure & Networking Software And Drivers
#1

I made a simple test for using more then one sge which use Send/Recv and RC qp.

sever side post two send work request immediately and then start poll cq.

client side post one recv work request then start poll cq, when recv ready, it then post another recv work request.

In my laptop (openSUSE Tumbleweed with rxe enabled) it runs as expected, but when I put the code run on server ( connect directly to mellanox device), after first work request was completed, there will be about 2 seconds delay, then the second work request complete. that’s weird.

the following picture is the full actions I’ve made

ksnip_20210514-145611
ksnip_20210514-1456111541×1673 725 KB
The code is in attachment.

How can I fix this problem ?

Thank you!

rdma.tar (60 KB)

#2

Hi,

I would suggest you to ask this question on linux-rdma mailing list as it seems to be a generic RDMA programming matter.

From brief review, it looks like there are mixed control/data patch. Application poll CQ and allocate resources in the same look that might be not optimal.

https://www.rdmamojo.com/2013/06/08/tips-and-tricks-to-optimize-your-rdma-code/#Avoid_using_control_operations_in_the_data_path

Try to continue with linux-rdma list, as this community is dedicated to specific Mellanox/Nvidia related questions.