Incorrect total memory reported by cudaMemGetInfo

Accelerated Computing CUDA CUDA Programming and Performance
1

Hello!

I’m getting incorrect total memory using cudaMemGetInfo function. I have 4 graphics cards: 3 of them (C2050) have 2.8 GB of global memory, and 1 (C2075) has 5.6 GB of global memory. When I use the C2075 card, the cudaMemGetInfo function reports only 2.8 GB of global memory. When I use the “totalGlobalMem” device property, it correctly reports 5.6 GB of global memory. Does anyone know why cudaMemGetInfo might be reporting incorrect total memory?

Many thanks!

2

Could you post some code?

3

Thanks for your response! Here’s the code:

cudaDeviceProp deviceProp;

cudaGetDeviceProperties(&deviceProp, 2);

long gpuGlobalMem = deviceProp.totalGlobalMem;

fprintf(stderr, "GPU global memory = %ld Bytes\n", gpuGlobalMem); /* Outputs 5,636,554,752 bytes. */

size_t freeMem, totalMem;

cudaMemGetInfo(&freeMem, &totalMem);

fprintf(stderr, "Free = %ld, Total = %ld\n", freeMem, totalMem); /* Outputs 2,817,982,464 bytes of total memory. */

I appreciate your help!

4

(1) Is the system running a 32-bit OS?
(2) Is the system running Windows, using the default WDDM driver (if so, what’s the amount of system memory on this platform) ?
(3) What is the version of CUDA and the version of the driver package ?

5

Hi njuffa,

(1) The system is running a 64-bit OS

(2) The OS is CentOS 5.4. The system memory is ~48 GB.

(3) CUDA toolkit version is 4.2, and the driver version is 295.53.

Thanks for your help!

6

Make sure none of the API calls returns an error. The size parameters are all size_t, so I would suggest storing and printing them accordingly. Does the output change if you use the following program?

#include <stdio.h>

#include <stdlib.h>

// Macro to catch CUDA errors in CUDA runtime calls

#define CUDA_SAFE_CALL(call)                                          \

do {                                                                  \

    cudaError_t err = call;                                           \

    if (cudaSuccess != err) {                                         \

        fprintf (stderr, "Cuda error in file '%s' in line %i : %s.\n",\

                 __FILE__, __LINE__, cudaGetErrorString(err) );       \

        exit(EXIT_FAILURE);                                           \

    }                                                                 \

} while (0)

int main (void)

{

    cudaDeviceProp deviceProp;

    CUDA_SAFE_CALL (cudaGetDeviceProperties(&deviceProp, 2));

    size_t gpuGlobalMem = deviceProp.totalGlobalMem;

    fprintf(stderr, "GPU global memory = %zu Bytes\n", gpuGlobalMem); 

    size_t freeMem, totalMem;

    CUDA_SAFE_CALL (cudaMemGetInfo(&freeMem, &totalMem));

    fprintf(stderr, "Free = %zu, Total = %zu\n", freeMem, totalMem); 

    return EXIT_SUCCESS;

}

If you still see wildly mismatching sizes reported with the code above, I would suggest filing a bug.

7

I ran the code you posted, but it still shows discrepancy:

GPU global memory = 5636554752 Bytes

Free = 2748571648, Total = 2817982464

I guess I need to file a bug report.

8

The driver team suggests to cycle (i.e. unload and reload) the kernel driver with

rmmod nvidia && modprobe nvidia

so you are working with a freshly loaded driver. We can’t think of anything that would cause the total memory size reported to differ between the two API calls, so a bug report seems in order. Sorry for the inconvenience. Bug reports can be filed via the registered develoepr website, as follows:

(1) Go to http://developer.nvidia.com/

(2) Click on the green link “Registered Developers Website” in the upper right corner

(3) Log in with your user name / password

(4) Click on the green link “CUDA/GPU Computing Registered Developer Program”

(5) Under the section heading “Reporting an Issue” click on the green link “The Submit a Bug Form”

(6) Please provide all requested information (detailed system configuration might really help in this case) and attach your repro code

Thank you for your help.

9

I just realized what the problem is. I had forgotten to set the device to 2 before calling the cudaMemGetInfo function. For this reason, it was reporting the total memory from the first GPU card, which only has 2.8 GB of global memory, instead of the 3rd GPU card, which has 5.6 GB of global memory. Now that I’m setting the correct GPU before calling cudaMemGetInfo function, I’m getting the correct results!

Thanks for your help!