When will an ACK generated in RDMA write?

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hi,

Recently, I am doing some RDMA write latency test with ConnectX-4 Lx 25G NIC.

And I have two questions about the testing:

  1. What is the version of RDMA? Is the default RoCEv2?
  2. By default, the RC(reliable) mode is chosen. As required, an ACK is needed from the remote to local, then the local will add an entry to the CQ and the software then know the data has arrived at the remote. I searched with google but didn’t find when a MLNX NIC will generate the ACK. I mean, is it generated when the data received by the HCA or after DMA to the host memory? And I assume the ACK is automatically generated by the NIC adapter without any SW involvement. Is my understanding right?

Many thanks

2

Hi,

In general, the question is not related to Mellanox, but more to Infiniband SPEC and linux-rdma mailing list could be a better choice. And seems that you already discussing the subject using another forum, which is great by the way - https://www.rdmamojo.com/2013/02/15/ibv_poll_cq/ https://www.rdmamojo.com/2013/02/15/ibv_poll_cq/

3

Hi Alkx,

Many thanks.

I will read the specification later when time is available. Yes, I did ask the question on that blog. When I click to commit my question, nothing update there. So I assumed that I failed to commit due to the network or the server’s problem. To my understanding, the NIC will generate the ACK once it received the whole frame/segment without any error to accelerate the processing. Indeed, almost all the chips will do so in the embedded platform, maybe the NIC on the server will have the same consideration.

I am just confused the cycles gap of poll cq between the RC and UC mode. And by the way, where can I get the latency and BW report of MLNX NICs? Is it public? From the blog, it is said that usually less than 1 microsecond with small packets. But my testing result is more than 1 us. And no matter whether the numactl is used to specify the CPU core and memory allocation.

4

Could any expert give a hand?

5

Hi Bing,

If it can help, here is the another link - http://www.cs.unh.edu/~rdr/rdma-intro-module.ppt http://www.cs.unh.edu/~rdr/rdma-intro-module.ppt - that shows that ACK generated by remote side when it write to HCA. It doesn’t guarantee that user-space application has it, but it is on remote side.

For any other question, it is preferably to ask a new question if it is specific to Mellanox, and for all other issues better to discuss on linux-rdma mailing list.

6

Hi Alkx,

Got it. Many thanks.