How to use Doorbell or RDMA Write with Immediate for data communication between two servers?

Hello Mellanox Support,

I have a task to write a data communication code , to send data from server A to server B, through IB (connextc-3 PRO). After data is sent from Server A to Server B, a notification should be sent to Server B, such that Server B can proceed to process/consume the data.

I plan to use door bell for this notification as I think the doorbell could consumes less CPU cycles at the Server B side.

Do you have any example code/sample code to use doorbell along with the IB programming? I have studied about perftest(ib_write, ib_test) code, but there is no such example for doorbell.

Can I also use RDMA Write with Immediate feature for this? Any sample code for this? what is the verb I should set to use RDMA Write with Immediate?

I also looked at the rdma_get_recv_comp() code in rdma_server/client.c example. as per manual, rdma_get_recv_comp() is blocking, does this “blocking” refer to block on CPU, or block on that specific CQ channels(but I can still use other CQ for data communication)?

Thank you.

Mei Guodong

Hi Guodong Mei,

that doorbell is ibv_post_send with opcode “IBV_WR_SEND”.

The ib_send_bw performance test tool is an example for this.

Another example is the BeeGFS parallel file system code, which does a pretty good job of abstracting the RDMA communication details to a Berkeley socket style interface with receiver notifications: http://www.beegfs.io http://www.beegfs.io/

Best regards,

Sven Breuner