To begin with,I’ve been playing RDMA with ConnectX-3 ConnectX3-Pro and ConnectX5 Ethernet Adapter for half a year.
The following reference documents provide me with much knowledge and help.
- RDMA_Aware_Programming_User_Manual.pdf (Version 1.7)
- the IB Specification Vol 1-Release-1.3-2015-03-03.pdf
But when I learn more,I want to know more about of the work principle of NIC.
The question comes for me is that "When NIC posts work requests from QueuePair,what information does it need? Does Queuepair information need to be packed with work request?"
Let’s review the procedure of NIC posting work request.
- UserSpace procedure:
- Create a QueuePair and set relative attribute and transit it to RTS state
- Post a send work request or recv work request to send queue or receive queue respectively
- DriverSpace procedure:
- Doorbell on the NIC got notification and retrive the work request by DMA from the Memory to NIC cache
- Pack the data with protocol header and tail and fragment the datagram into packet to send into cabel
I don’t know what happened between the UserSpace procedure and DriverSpace procedure.
Does NIC also fetch the queuepair information to the NIC cache for necessary information to fill in packets?
I try to search for some reference to find out the real procedure,but haven’t got enough proof.
But some information can provide some facts as follow:
In Connected Transport type,such as Reliable Connections(RC):
- A local queuepair connects to a remote queuepair
- The work request contains the field as follow:
As the work request seems to contain enough information to be recognized by local NIC and remote NIC,
does it need to collect information about QueuePair?
Can some predecessors give me some help?Very thanks！