How to know if GPUDirect is installed/supported GPUDirect

Accelerated Computing CUDA CUDA Programming and Performance
1

Hello,

We have a two-node system each with a Tesla C2050 and a Mellanox MT26428 QDR Infiniband card, and we’d like to test out the capabilities of GPUDirect version 1.0 (where the IB card and CUDA can share pinned pages). I compiled the “mpi_pinned.c” application (from the nvidia-gpudirect-3.2-1.tar.gz tarball) as a cuda app (change to mpi_pinned.cu and compile with nvcc) and have run it with OpenMPI, but it’s unclear as to whether GPUDirect v1.0 is installed or whether it is even supported for our setup. This earlier forum link seems to indicate that CUDA 4.0 would remove the need to install the patches in the nvidia-gpudirect-3.2-1 tarball on the main GPUDirect website.

Does anyone know if CUDA 4.0 still requires patches to the OS and IB driver to use GPUDirect v1.0? Also, is there any easy way to check and see if GPUDirect 1.0 is properly set up and configured?

Thanks,
Jeff

System setup:

ArchLinux kernel version 3.0, 64-bit
OFED 1.5 installed
OpenMpi 1.5.4 used to run MPI applications.
NVIDIA CUDA SDK 4.0
NVIDIA driver 280.13

2

4.0: you don’t need the patch if you’re willing to set the environment variable CUDA_NIC_INTEROP to 1 and you’re running at least 2.6.18. That will enable the magic new allocation path which removes the need for the kernel patch and special OFED drivers–you get all of the performance of the patch for free.

4.1 (coming soon to a CUDA download page near you): no more kernel patch, new allocation path is enabled by default on kernels >= 2.6.18.