How to investigate "Resource exhausted: OOM when allocating tensor"?

[at589]

Hello,
I’m playing with tiny text generation model on my Jetson Nano (JetPack 4.6 [L4T 32.6.1] ), and sometimes I get this exception

2022-07-24 05:03:49.700418: I tensorflow/core/common_runtime/bfc_allocator.cc:1058] Sum Total of in-use chunks: 155.25MiB
2022-07-24 05:03:49.700449: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] total_region_allocated_bytes_: 162787328 memory_limit_: 162787328 available bytes: 0 curr_region_allocation_bytes_: 325574656
2022-07-24 05:03:49.700489: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Stats:
Limit:                       162787328
InUse:                       162787328
MaxInUse:                    162787328
NumAllocs:                        7492
MaxAllocSize:                 22020096
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2022-07-24 05:03:49.700980: W tensorflow/core/common_runtime/bfc_allocator.cc:467] ************************xxxx************************************************************************
2022-07-24 05:03:49.701082: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at split_op.cc:308 : Resource exhausted: OOM when allocating tensor with shape[32,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

As far as I understand, there was not enough memory, but the following is unclear

1. What kind of memory, GPU or RAM?
2. What took all the memory, data or model layers?
3. Is it possible to evaluate the amount of needed memory before the fitting operation? How?

May be there is a step-by-step guide about memory allocation and memory amount evaluation for Nvidia chips?

Thank you!

[AastaLLL]

Hi,

Jetson is a shared memory system. CPU and GPU share the physical memory.
Based on the error, the memory is exhausted when allocating a tensor with shape=[32,256].

Thanks.