Consumption of host memory increases abnormally

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,

I just found that the consumption of host memory seems to increase abnormally in the new CUDA release.
The following is a very simple demo program.

int main()
{
// set GPU ID
cudaSetDevice( 0 );

// allocate host memory
const long NGB = 9;
const long N = NGB10241024*1024/sizeof(float);
float *Array = new float [N];

// arbitrary work on Array
// …

delete Array;
exit(0);
}

The “top” command shows that the demo program consumes about 18.3 GB, which is about 2 times larger than expected!!
However, if i comment out the line “cudaSetDevice( 0 );”, the memory consumption becomes normal (about 9 GB).
The information of the system giving this issue is as follows.

CUDA version: 4.0 RC2
Driver version: 270.41.06
OS: Scientific Linux SL release 5.4 (Boron)

I suspect that this issue is due to the new graphic driver and(or) CUDA since this problem appeared after a system upgrade.
I also tested the demo program in a different system with older versions of CUDA and driver, and everything works fine.
In this case, the memory consumption is normal even when I include any CUDA function.
The information of this system is as follows:

CUDA version: 3.2
Driver version: 260.19.21
OS: CentOS release 5.5 (Final)

Has anyone had a similar problem? Any help is appreciated!

2

I’m pretty sure you’re looking at the total memory column, and this is irrelevant.

The RSS column is what your application really uses, and I don’t think this one will change dramatically with just the addition of cudaSetDevice().

You can try “pmap -x ” to see where your memory is.

Same thing if you mmap a 200GB file (which sits on disk). You would see 200GB in top. That does not mean you’re using 200GB.

-Guillaume

3

Thomasco, thanks a lot for your reply External Image
You are right. Here I refer to the “VIRT” memory shown by the top command.
The actual memory consumption (the RSS column shown by “pmap -x”) is normal.
However, abnormal virtual memory consumption is still an unpleasant property.
Especially, in the system with a limited amount of virtual memory, the program
can crash due to the enormous consumption of virtual memory.

I have verified that the abnormal virtual memory consumption only occurs in CUDA 4.0
(in both RC2 and the production release).
If I reinstall both driver and CUDA toolkit to version 3.2, this issue no longer appears.

Does anyone has any suggestion?

4

I also noticed this on my seven GPU testbed with CUDA 4.0. Once I open a CUDA context on a device, the process grabs 24 GB of virtual memory. The actual memory usage is fine, so there is no reason to be worried, but it is a little weird.

Is this related to the Unified Virtual Addressing feature of CUDA 4?

5

this is related to UVA. we have to carve out a chunk of virtual memory equal to the total physical GPU memory, plus the total system memory, plus some small fudge factor for alignment purposes.

we actually throttle back on the UVA region if you run out of virtual memory. this will restrict the amount of memory you can allocate, though.

6

Tmurray, thanks for the explanation External Image
So, do you have any suggestion for the case that the program is terminated due to
the large amount of virtual memory consumption (which exceeds the upper limit
reported by “ulimit”)?
Is there any solution to get rid of this issue?
Otherwise, I will have to try to convince the system administrator to reset the upper limit
of virtual memory to “unlimited”.