I tried to use GPUDirect between Quadro p4000 and 1080 Ti, but SimpleP2P showed me p2p between these two GPUs is NO.
ubuntu@:~$ nvidia-smi topo -m
GPU0 GPU1 GPU2 CPU Affinity
GPU0 X PHB SOC 0-31
GPU1 PHB X SOC 0-31
GPU2 SOC SOC X 0-31
Legend:
X = Self
SOC = Connection traversing PCIe as well as the SMP link between CPU sockets(e.g. QPI)
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
PIX = Connection traversing a single PCIe switch
NV# = Connection traversing a bonded set of # NVLinks
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 3
> GPU0 = "GeForce GTX 1080 Ti" IS capable of Peer-to-Peer (P2P)
> GPU1 = " Quadro P4000" IS capable of Peer-to-Peer (P2P)
> GPU2 = " Quadro P4000" IS capable of Peer-to-Peer (P2P)
Checking GPU(s) for support of peer to peer memory access...
> Peer access from GeForce GTX 1080 Ti (GPU0) -> Quadro P4000 (GPU1) : No
> Peer access from GeForce GTX 1080 Ti (GPU0) -> Quadro P4000 (GPU2) : No
> Peer access from Quadro P4000 (GPU1) -> GeForce GTX 1080 Ti (GPU0) : No
> Peer access from Quadro P4000 (GPU1) -> Quadro P4000 (GPU2) : No
> Peer access from Quadro P4000 (GPU2) -> GeForce GTX 1080 Ti (GPU0) : No
> Peer access from Quadro P4000 (GPU2) -> Quadro P4000 (GPU1) : No
Two or more GPUs with SM 2.0 or higher capability are required for ./simpleP2P.
Peer to Peer access is not available amongst GPUs in the system, waiving test.
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.