I am having the following problem with my multi-GPU setup. The CUDA simpleP2P does not work:
[./simpleP2P] - Starting... Checking for multiple GPUs... CUDA-capable device count: 2 > GPU0 = " Tesla K40c" IS capable of Peer-to-Peer (P2P) > GPU1 = " Tesla K40c" IS capable of Peer-to-Peer (P2P) Checking GPU(s) for support of peer to peer memory access... > Peer-to-Peer (P2P) access from Tesla K40c (GPU0) -> Tesla K40c (GPU1) : No > Peer-to-Peer (P2P) access from Tesla K40c (GPU1) -> Tesla K40c (GPU0) : No Two or more GPUs with SM 2.0 or higher capability are required for ./simpleP2P. Peer to Peer access is not available between GPU0 <-> GPU1, waiving test.
I have checked that the two cards are on the same PCI-E root, and I think they are:
$ lspci | grep NVIDIA 81:00.0 3D controller: NVIDIA Corporation GK110BGL [Tesla K40c] (rev a1) 82:00.0 3D controller: NVIDIA Corporation GK110BGL [Tesla K40c] (rev a1) $ lspci -t .. .. +-[0000:80]-+-02.0-----00.0 | +-03.0-----00.0 | +-05.0 | +-05.1 | +-05.2 | \-05.4 ... ...
Also, nvidia-smi is telling me that the GPUs communicate with PCI-E bridge:
$ nvidia-smi topo -m GPU0 GPU1 CPU Affinity GPU0 X PHB 8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31 GPU1 PHB X 8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31 Legend: X = Self SOC = Path traverse a socket-level link (e.g QPI) PHB = Path traverses a PCIe host bridge PXB = Path traverses multiple PCIe internal switches PIX = Path traverses a PCIe internal switch
I am running the system with Fedora 20, with the official CUDA instalation. All other examples work. Any ideas why peer to peer memory access does not work ?