Inconsistent GPU memory usage between Cuda 7.5 and Cuda 8.0

Accelerated Computing CUDA CUDA Programming and Performance
1

I have recently upgraded the product I have been working on from CUDA 7.5 to CUDA 8.0.

Before the upgrade my memory usage would start at around 2.5 GB of GPU memory total after my first set of CUDA kernels have run(Windows is using about 1.2 GB of GPU memory before I even start the application).

After the upgrade my memory usage would immediately jump to > 5GB of GPU memory after the first set of CUDA kernels.

Is this new memory usage expected with the upgrade to CUDA 8.0? Its quite possible I have done something wrong and the issue is exposing itself after the upgrade. Is there any gotchas that I may have missed causing this sudden spike in usage between the versions?

Other Information:

GPU Hardware: GTX 1080
Driver Version: 369.30
NVCC Build FLAGS: //Note: I have tried increasing the compute version to only build compute_61
’ --gpu-architecture=compute_30’
’ --gpu-code=sm_30,sm_35,sm_50,sm_52’
’ --generate-code arch=compute_52,code=compute_52’
’ --machine=64’
Relevant MSVC flags: /MD

2

Is this data from a controlled experiment (meaning: same physical machine, same GPU, the only difference is that CUDA was updated from version 7.5 to version 8.0)?

3

Yes the only difference is the upgrade from CUDA 7.5 to CUDA 8. Everything else is the same.

However I have run the test on another machine as well which has a 980 GTX and see the same issue with different memory usages.

4

try updating to the latest driver for GTX 1080

5

Ill try the latest drivers when I have time, rebuilds take a decent amount of time. I will be very surprised if its the drivers though as I saw this issue with the RC as well and have tried updating my drivers several times since then.

6

it looks like switching from 32-bit to 64-bit. f.e. you previously used -m32, or library you are using switched from using unsigned to size_t to ensure that it can deal with arbitrary input. if you have some “main array”, try to figure out its element type, and where it is defined