W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at pack_op.cc:88 : Resource exhausted: OOM when allocating tensor with shape[32,3,2160,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Accelerated Computing Intelligent Video Analytics TAO Toolkit
16

This should be case by case.
For your case, you are training with GTX 1070 but meet OOM for (4096,2160).
If you use multi-gpu or other GPU, the OOM issue maybe does not happen.

So, please keep you hardware setting and resize the images/labels, to see if OOM issue is gone.

1 Like