Cuda Out of Memory with tons of memory left?

Accelerated Computing CUDA CUDA Programming and Performance
1

When running my CUDA application, after several hours of successful kernel execution I will eventually get an out of memory error caused by a CudaMalloc.

However, when I check the memory remaining I have over 400MB free, and the CudaMalloc call itself is a float array allocation of no more than 5000 elements.

Is it possible this is not actually “out of memory” but some memory has been corrupted? I am not sure if this is related to this [url=“http://support.microsoft.com/kb/940105”]http://support.microsoft.com/kb/940105[/url] but I am running XP, not Vista does this even issue even apply?

Any advice on how to debug this would also be greatly appreciated, but running in emulation mode appears not to recreate the problem.

Thanks.

2

Are you sure you are checking the memory available on the graphics card? (How do you do that? I am not aware of mechanisms for this, though I am just a beginner in CUDA as well)

Perhaps you must have a memory leak somewhere in the cuda related portion of your code?

It sounds like the memory allocation for memory within the graphics card is failing due to lack of memory…

3

I am using cuMemGetInfo to get the available memory, basically I run the code for several hours and the memory is all freed perfectly fine. Then, the amount of free memory increases, and the code still seemingly works fine for about another 30 mins or so. Then the code fails with “out of memory” on the second call to CudaMalloc in my function that allocates all the memory.

4

DalekCuda, could you finally solve this problem? I happen to have exactly the same problem… cuMemGetInfo() tells me that I still have some 91Mb of free memory (out of a total of 267MB) and when I try to allocate ca. 50Mb it crashes. I have a Quadro FX360M under Ubuntu 8.10 (using SDK 2.1)

Thanks for your help in advance!

5

Hi,
I have had similar issues in the past, and you have two reasons why this will happen.
I work mainly with Matlab and cuda, and have found that the problem of Out of Memory given in Matlab while executing a CUDA MexFile is not allways caused by CUDA being out of memory, but because of Matlab and the CPU side being without memory.
So it is recommended to use the /3GB comand in the boot.ini for winXP systems, and tha appropriate commando for Vista systems.
If you try the Matlab function memstats, you will see the improvement in memory. Even if you are not using memory, the idea that i am trying to put forward is that an out of memory while executing CUDA is not necessarily because of cuda being out of memory.
So please try the 3GB command to amplify memory of system, or make the pageable memory larger.
For more refference in Matlab, you will find this post interesting:
[url=“http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/brh72ex-49.html”]http://www.mathworks.com/access/helpdesk/h...brh72ex-49.html[/url]
The other reason of an out of memory is that you are not free-ing all the reserved memory. I would recomment counting all cudamallocs, and then count cudafree to make sure they are even.
Hope this helps, and kind regards,
David Lisin

6

Thank you very much for your help. I will definitely try the approach you recommend.

Thanks again.