First using the gpu-direct RDMA


I’m trying to learn how to use GPU-direct RDMA.

I connect ConnectX3-Pro to Jetson AGX Xavier, then plug the ethernet ports of the ConnectX to each other in order to create a loop.
I want to write simple program that do the following steps:

  1. Creates a data-buffer in the GPU memory.
  2. Copies the data to another buffer in the GPU memory using the GPU-direct RDMA
  3. Compares the two buffers and check if the data-transfer worked properly

I have two questions:

  1. Is the configuration of the system I described above is legit? It can work?
  2. I found in the GPUdirect-RDMA documentation information about the GPU-direct mechanism, and about the operations I need to perform before and after the data-transfer (pinning, unpinning) . But I didn’t found the commands of the transfer itself. Lets say I created the first buffer and pinned the memory region of the buffer, how to transfer the data using the RDMA? APIs of some kind? Is there some simple example of such program I can use?

Thank you in advance,