Getting OOM when using large images with image on same model whereas small images work fine.
Device: Jetson Nano
Jetpack: 4.2.1 L4T 32.2.0
Mem: 4.0GB
Swap 16.3GB
TF: 1.15.0
This is recognition snippet
img_path = './1.png'
# %matplotlib inline
import matplotlib.pyplot as plt
set_session(tf_sess)
# Get a set of three example images
img = image.load_img(img_path, target_size=image_size[:2])
img = cv2.cvtColor(np.float32(img), code=cv2.COLOR_RGB2GRAY)[..., np.newaxis]
img = img.astype('float32') / 255
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
#x = preprocess_input(x)
from keras import backend as K
K.clear_session()
K.set_learning_phase(0)
with tf.get_default_graph().as_default():
set_session(tf_sess)
feed_dict = {
input_tensor_name: x
}
preds = tf_sess.run(output_tensor, feed_dict)
# decode the results into a list of tuples (class, description, probability)
# (one such list for each sample in the batch)
print('Predicted:', preds)
`2020-04-22 12:27:40.190706: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 9437184 exceeds 10% of system memory.
2020-04-22 12:27:40.194065: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 9437184 exceeds 10% of system memory.
2020-04-22 12:27:40.398451: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 9437184 exceeds 10% of system memory.
2020-04-22 12:27:40.401647: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 9437184 exceeds 10% of system memory.
2020-04-22 12:27:40.522248: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 9437184 exceeds 10% of system memory.
2020-04-22 12:27:52.996295: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.50MiB (rounded to 4718592). Current allocation summary follows.
2020-04-22 12:27:52.997559: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256): Total Chunks: 1, Chunks in use: 1. 256B allocated for chunks. 256B in use in bin. 256B client-requested in use in bin.
2020-04-22 12:27:52.998471: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512): Total Chunks: 1, Chunks in use: 1. 512B allocated for chunks. 512B in use in bin. 512B client-requested in use in bin.
2020-04-22 12:27:52.998896: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024): Total Chunks: 2, Chunks in use: 2. 2.0KiB allocated for chunks. 2.0KiB in use in bin. 2.0KiB client-requested in use in bin.
2020-04-22 12:27:52.999099: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048): Total Chunks: 1, Chunks in use: 1. 2.2KiB allocated for chunks. 2.2KiB in use in bin. 2.2KiB client-requested in use in bin.
2020-04-22 12:27:52.999299: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:52.999504: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:52.999702: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:52.999918: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.000107: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.000345: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.000608: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144): Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2020-04-22 12:27:53.000824: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288): Total Chunks: 1, Chunks in use: 0. 731.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.001040: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.001250: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152): Total Chunks: 1, Chunks in use: 1. 2.00MiB allocated for chunks. 2.00MiB in use in bin. 1.12MiB client-requested in use in bin.
2020-04-22 12:27:53.001484: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304): Total Chunks: 1, Chunks in use: 1. 4.00MiB allocated for chunks. 4.00MiB in use in bin. 2.25MiB client-requested in use in bin.
2020-04-22 12:27:53.001681: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.001893: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.002267: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.002450: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.002564: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.002665: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-04-22 12:27:53.002763: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 4.50MiB was 4.00MiB, Chunk State:
2020-04-22 12:27:53.002933: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1048576
2020-04-22 12:27:53.003078: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0xf00870000 next 1 of size 2304
2020-04-22 12:27:53.003211: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0xf00870900 next 2 of size 256
2020-04-22 12:27:53.003609: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0xf00870a00 next 3 of size 294912
2020-04-22 12:27:53.003776: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0xf008b8a00 next 4 of size 512
2020-04-22 12:27:53.003860: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0xf008b8c00 next 6 of size 1024
2020-04-22 12:27:53.003953: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0xf008b9000 next 8 of size 1024
2020-04-22 12:27:53.004095: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0xf008b9400 next 18446744073709551615 of size 748544
2020-04-22 12:27:53.004193: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 2097152
2020-04-22 12:27:53.004389: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0xf00970000 next 18446744073709551615 of size 2097152
2020-04-22 12:27:53.004628: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 4194304
2020-04-22 12:27:53.004814: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0xf00b70000 next 18446744073709551615 of size 4194304
2020-04-22 12:27:53.004968: I tensorflow/core/common_runtime/bfc_allocator.cc:914] Summary of in-use Chunks by size:
2020-04-22 12:27:53.005133: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 256 totalling 256B
2020-04-22 12:27:53.005310: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 512 totalling 512B
2020-04-22 12:27:53.005459: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1024 totalling 2.0KiB
2020-04-22 12:27:53.005552: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2304 totalling 2.2KiB
2020-04-22 12:27:53.005639: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 294912 totalling 288.0KiB
2020-04-22 12:27:53.005731: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 2097152 totalling 2.00MiB
2020-04-22 12:27:53.005841: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 4194304 totalling 4.00MiB
2020-04-22 12:27:53.005910: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 6.29MiB
2020-04-22 12:27:53.006009: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 7340032 memory_limit_: 8441856 available bytes: 1101824 curr_region_allocation_bytes_: 8388608
2020-04-22 12:27:53.006147: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats:
Limit: 8441856
InUse: 6591488
MaxInUse: 6591488
NumAllocs: 8
MaxAllocSize: 4194304
2020-04-22 12:27:53.006552: W tensorflow/core/common_runtime/bfc_allocator.cc:424] _________xxxxxxxxxxx***********xxxxxxxxxxxxxxxxxxxxxxxxx
2020-04-22 12:27:53.006769: W tensorflow/core/framework/op_kernel.cc:1628] OP_REQUIRES failed at constant_op.cc:77 : Resource exhausted: OOM when allocating tensor of shape [3,3,256,512] and type float
2020-04-22 12:27:53.006926: E tensorflow/core/common_runtime/executor.cc:648] Executor failed to create kernel. Resource exhausted: OOM when allocating tensor of shape [3,3,256,512] and type float
[[{{node conv_5/Conv2D/ReadVariableOp}}]]
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1365, in _do_call
return fn(*args)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1350, in _run_fn
target_list, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [3,3,256,512] and type float
[[{{node conv_5/Conv2D/ReadVariableOp}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “predict.py”, line 84, in
preds = tf_sess.run(output_tensor, feed_dict)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 956, in run
run_metadata_ptr)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1180, in _run
feed_dict_tensor, options, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1359, in _do_run
run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [3,3,256,512] and type float
[[node conv_5/Conv2D/ReadVariableOp (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Original stack trace for ‘conv_5/Conv2D/ReadVariableOp’:
File “predict.py”, line 27, in
tf.import_graph_def(trt_graph, name=‘’)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/importer.py”, line 405, in import_graph_def
producer_op_list=producer_op_list)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/importer.py”, line 517, in _import_graph_def_internal
_ProcessNewOps(graph)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/importer.py”, line 243, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 3561, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 3561, in
for c_op in c_api_util.new_tf_operations(self)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 3451, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 1748, in init
self._traceback = tf_stack.extract_stack()
`