CNN model too big for GPU's dedicated memory

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,
I’m running a TensorFlow CNN model, within a Unity app, using the scisharp.tensorflow.redist-windows-gpu library. (TF: 2.7, CUDA: 11.2, cuDNN 8.1)
When running the app with the CNN model, the performance monitor shows around 5Gb usage of dedicated GPU memory. When running the same app on a computer with a graphic card with 4Gb, I get the following error message:

 (0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[300,1024,28,28] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Is there any way to run this app on the computer with the smaller GPU memory? for example by using the shared GPU memory, or any other idea?

Thanks

2

Hi denise,

The easiest way to reduce memory usage would be to reduce the batch size when you are trying to run the model.

This will reduce the throughput of the model, but you can fine tune it so that it uses as much memory before getting the OOM error.

This modification will be either in the specific function you are using to configure the model or in the actual model when you are loading it.