How to use GPU Direct RDMA with infiniband ConnectX-4?

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

I am having trouble setting up GPU Direct on the local machines. Here’s the the local software and hardware:

  • GPU Tesla P100-SXM2
  • Adaptor（MLNX）
    5e:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
    5e:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
  • Cuda compilation tools, release 10.1, V10.1.243
  • Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-167-generic x86_64)

I tested the RDMA connection by using ibping, and it works fine.

--- anton-j0.(none) (Lid 2) ibping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 1030 ms
rtt min/avg/max = 0.005/0.103/900.020 ms

However, when I was trying to get GPU Direct RDMA to run, nv_peer_mem wouldn’t install. And as the github demo indicated, it requires ConnectX 5+ to work.

I tried to find other ways that is compatible with ConnectX 4 but hasn’t found anything useful yet. I checked the forum and someone got ConnectX 3 pro to work on GPU Direct RDMA. Could someone give me some guidelines to get GPU Direct RDMA working on ConnectX 4?

2

Hello and thank you for writing us.
This issue can be happening due to some issues.
GPU Direct RDMA is supported with any NVIDIA ConnectX-4 (or later) InfiniBand adapter card. This means that your ConnectX-4 adapters should be compatible with GPU Direct RDMA.
I would like to advice on opening a case in our Enterprise service Portal
This will allow us to dig dipper in to the Issue and help.

Thanks and have a great day!
Ilan.