Resource exhausted: OOM when allocating tensor with shape[256]

[ssuresh.mca]

I am getting OutOfMemory exception, How to resolve this issue?

Python version :3.6
Jetson Nano Jetpack version: 4.4
Tensor flow version: tensorflow==1.15.2+nv20.6
Keras: 2.0.5
I have attached the source code as image file

Error :

tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Source code

Refer the Logs, generated while start executing

“adding visible gpu devices: 0”

is GPU not allocated for the job?

[AastaLLL]

Hi,

Could you check if the configure shared in this topic helps or not first?

Resource Exhausted Error

Hi, Could you try to set some limitation on the TensorFlow memory usage to see if helps first? 1. Limit the maximal memory available for TensorFlow gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.6) sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options)) 2. Allow gradually memory allocation config = tf.ConfigProto() config.gpu_options.allow_growth=True sess = tf.Session(config=config) Thanks.

Thanks.

[ssuresh.mca]

I have improved the performance by specifying configs per_process_gpu_memory_fraction =0.2, allow_growth =True and visible_device_list=“0”. But at one point in time, Swap memory reaches “100%” and System freezed. sometimes System give below error message as shown in the screenshot.

Could you suggest to resolve this issue?

[linuxdev]

You could increase the size of swap, but that only goes so far since the GPU needs physical RAM. Other processes can be swapped, but if the actual GPU requires more RAM, then you’d need your code to be rearranged to use less memory.

[ssuresh.mca]

I have tried hard limit on the GPU. I have created swap file of size 6 GB. But again i got Out of memory exception on Jetson Nano.
I have restricted TensorFlow to only allocate 2560 MB (memory_limit=2560) of memory on the first GPU
I am using Stanford Cars Dataset

this is the source code

Please help me to solve this issue

[linuxdev]

Your approach is correct, but I don’t know enough about the AI side to provide useful methods of limiting memory for your case. Someone else will probably have a way to reduce the required memory, but this is a typical reason to use something which has more RAM, e.g., the NX is the next step up and it has 8GB of RAM (the NX is a newer GPU architecture and six cores as well, and seems to work on the same carrier board a Nano b02 or newer would use).