Trying to get GPUdirect RDMA working.


I am trying to figuere out this GPUdirect RDMA thing.
I am trying to get the p2pToken and vaSpaceToken for the GPU allocated memory.

int main(int argc, char **argv)
    CUresult ret;

    int deviceCount = 0;
    // Get number of devices supporting CUDA
    if (deviceCount == 0) {
        printf("There is no device supporting CUDA.\n");

    CUdevice cuDevice;
    cuDeviceGet(&cuDevice, 0);

    CUcontext cuContext;
    ret = cuCtxCreate(&cuContext, CU_CTX_MAP_HOST , cuDevice);
    if(ret != CUDA_SUCCESS) {
        printf("ERROR cuCtxCreate\n");
    char deviceName[256];
    cuDeviceGetName(deviceName, 256, cuDevice);
    printf("\nDevice %d: \"%s\"\n", cuDevice, deviceName);
    int major = 0, minor = 0;
    cuDeviceComputeCapability(&major, &minor, cuDevice);

    int driverVersion = 0;
    printf("  CUDA Driver Version:                           %d.%d\n", driverVersion/1000, (driverVersion%100)/10);
    printf("  CUDA Capability Major/Minor version number:    %d.%d\n", major, minor);
    int unifiedAddressing;
    cuDeviceGetAttribute(&unifiedAddressing, CU_DEVICE_ATTRIBUTE_UNIFIED_ADDRESSING, cuDevice);
    printf("  Device supports Unified Addressing (UVA):      %s\n", unifiedAddressing ? "Yes" : "No");
    int pciBusID, pciDeviceID;
    cuDeviceGetAttribute(&pciBusID, CU_DEVICE_ATTRIBUTE_PCI_BUS_ID, cuDevice);
    cuDeviceGetAttribute(&pciDeviceID, CU_DEVICE_ATTRIBUTE_PCI_DEVICE_ID, cuDevice);
    printf("  Device PCI Bus ID / PCI location ID:           %d / %d\n\n", pciBusID, pciDeviceID);

    CUdeviceptr mem;
    ret = cuMemAlloc(&mem, 64*64);
    if (ret != CUDA_SUCCESS) {
        printf("ERROR cuMemAlloc\n");
    ret = cuPointerGetAttribute( &tokens, CU_POINTER_ATTRIBUTE_P2P_TOKENS, mem );
    if (ret == CUDA_SUCCESS) {
        printf("This is working\n");
    } else if (ret == CUDA_ERROR_INVALID_DEVICE)  {
    } else {
        printf("Error code %d\n", ret);

When I run this I get the output.

Device 0: “GeForce GTX 460”
CUDA Driver Version: 5.0
CUDA Capability Major/Minor version number: 2.1
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 1 / 0


What I do not understand is to the call cuPointerGetAttribute I get returned CUDA_ERROR_INVALID_DEVICE and not CUDA_SUCCESS. Am I doing something wrong or misunderstand something since I get CUDA_ERROR_INVALID_DEVICE or is this a setup issue?

I have another problem about GPUDirect 3.0 RDMA. According to this document

It seems only support Linux module driver. But how about Linux built-in driver.

How can I do if I want to modify built-in driver to support GPUDirect 3.0 RDMA ?

I know this issue was posted long time ago but was wondering if the cause of the issue was identified? I am currently facing the same issue. (Hardware/software information: Geforce 670 (and tried it on 650 as well); cuda 6.0 toolkit; nvidia-331.44 kernel driver; unified addressing capability; compute capability 3.0 supported by the card; Fedora 17 with linux 3.4.4.)