How to specify the procedure of ConnectX-series NIC sending work request

To begin with,I’ve been playing RDMA with ConnectX-3 ConnectX3-Pro and ConnectX5 Ethernet Adapter for half a year.

The following reference documents provide me with much knowledge and help.

  1. RDMA_Aware_Programming_User_Manual.pdf (Version 1.7)
  2. the IB Specification Vol 1-Release-1.3-2015-03-03.pdf

But when I learn more,I want to know more about of the work principle of NIC.

The question comes for me is that "When NIC posts work requests from QueuePair,what information does it need? Does Queuepair information need to be packed with work request?"

Let’s review the procedure of NIC posting work request.

  • UserSpace procedure:
    • Create a QueuePair and set relative attribute and transit it to RTS state
    • Post a send work request or recv work request to send queue or receive queue respectively
  • DriverSpace procedure:
    • Doorbell on the NIC got notification and retrive the work request by DMA from the Memory to NIC cache
    • Pack the data with protocol header and tail and fragment the datagram into packet to send into cabel

I don’t know what happened between the UserSpace procedure and DriverSpace procedure.

Does NIC also fetch the queuepair information to the NIC cache for necessary information to fill in packets?

I try to search for some reference to find out the real procedure,but haven’t got enough proof.

But some information can provide some facts as follow:

In Connected Transport type,such as Reliable Connections(RC):

  • A local queuepair connects to a remote queuepair
  • The work request contains the field as follow:

As the work request seems to contain enough information to be recognized by local NIC and remote NIC,

does it need to collect information about QueuePair?

Can some predecessors give me some help?Very thanks!

It seems that RDMA is not a topic here…

Hi Martijn van Breugel,

http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf

I got very useful information from the reference above.

It helped me capture more details about adapter’s inner structure and successfully get the answer from the provided reference.

And I found that the RDMAMojo is really a good site for obtaining RDMA programming knowledge…(Never heard about).

So very thank you for giving me so much help!

Hi Haonan Qui,

Thank you for contacting the Mellanox Community Support page.

Based on the information you have provide, this information you want is more a general RDMA Programming question.

The best starting point for this, is to download the Mellanox Adapters Programmer’s Reference http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf

The PRM explains the work principle of the NIC regarding the work requests.

Further information can be obtained be going through the sourcecode of the perftest tool, examples from libibverbs and librdmacm.

Also RDMAmojo http://www.rdmamojo.com/tag/posting-work-request/ has a lot of general examples related to RDMA programming.

Thanks and regards,

Mellanox Community Support