We have a two-node system each with a Tesla C2050 and a Mellanox MT26428 QDR Infiniband card, and we’d like to test out the capabilities of GPUDirect version 1.0 (where the IB card and CUDA can share pinned pages). I compiled the “mpi_pinned.c” application (from the nvidia-gpudirect-3.2-1.tar.gz tarball) as a cuda app (change to mpi_pinned.cu and compile with nvcc) and have run it with OpenMPI, but it’s unclear as to whether GPUDirect v1.0 is installed or whether it is even supported for our setup. This earlier forum link seems to indicate that CUDA 4.0 would remove the need to install the patches in the nvidia-gpudirect-3.2-1 tarball on the main GPUDirect website.
Does anyone know if CUDA 4.0 still requires patches to the OS and IB driver to use GPUDirect v1.0? Also, is there any easy way to check and see if GPUDirect 1.0 is properly set up and configured?
ArchLinux kernel version 3.0, 64-bit
OFED 1.5 installed
OpenMpi 1.5.4 used to run MPI applications.
NVIDIA CUDA SDK 4.0
NVIDIA driver 280.13