GPUDirect RDMA: Difference between ibv_reg_mr and ibv_reg_dmabuf_mr

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hey there,

I’m still trying to understand what is the actual difference between these two calls?
We have either

  1. ibv_reg_mr
    or
  2. ibv_reg_dmabuf_mr

But how does it impact the data flow between the GPU and NIC? Can you please explain it based on the PCI topology? From my current understanding, the GPU and NIC share a pinned memory block on host memory filled with virtual addresses through which they communicate with each other. Afaik this is the regular implementation for ibv_reg_mr when using GPU memory. But what exactly happens when using ibv_reg_dmabuf_mr?

dmabuf however is not supported by my GPU (RTX A5000) as noted by perftest > ib_write_bw via the following code:

cuDeviceGetAttribute(&is_supported, CU_DEVICE_ATTRIBUTE_DMA_BUF_SUPPORTED, cuDevice)

2

Hi ,

ibv_reg_mr is memory registration allocated on GPU with nvidia-peermem library usage and ibv_reg_dmabuf_mr is registration with DMAbuff usage

Further information about DMABUF:
https://kernel.org/doc/html/v5.18/userspace-api/media/v4l/dmabuf.html?highlight=dma%20buffer

For RDMA code :

Thanks,
Samer

3

Thank you very much.

Can you specify what are the pro/cons of dmabuf over dma? What should we prefer to use for GDR for best performance?

4

I would appreciate any help or hints on my open question