GPUdirect RDMA with NVIDIA A100 for PCIe

Accelerated Computing DGX User Forum
, ,
#1

Hi,
I have an NVIDIA A100 card connected to a server through PCIe gen 4. I want to transfer data directly from an FPGA card to the A100 for real-time data processing. If I transfer data from FPGA to CPU RAM (XDMA) and from RAM to GPU (cudaMemcpy) for processing, it would be very slow and does not meet my requirements. So I have the following questions:

  1. Is A100 support GPUdirect RDMA?
  2. Is it possible to transfer data directly from the FPGA to the A100 through GPUdirect RDMA?
  3. If RDMA is possible for A100, could you suggest to me some reference documents or anything related to RDMA so that I can follow and build my own design?

Thank you so much!!!

#2

Yes, A100 supports GDRDMA

You would need to write a kernel module for your FPGA.

RDMA related documentation can be found here:

Some examples:

The nv_peer_memory module example:

Jetson RDMA: