Whith great dificulty I was able to insatll all packages. Now im trying to run a mask rcnn .h5 file(250). Getting memory error

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

Use tf.cast instead.
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1356, in _do_call
return fn(*args)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1024,324] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node mrcnn_bbox_fc/random_uniform/RandomUniform}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/safran/Mask-RCNN-series-master/process_video.py”, line 3, in
from visualize_cv2 import model, display_instances, class_names
File “/home/safran/Mask-RCNN-series-master/visualize_cv2.py”, line 27, in
model.load_weights(COCO_MODEL_PATH, by_name=True)
File “/home/safran/Mask-RCNN-series-master/model.py”, line 2130, in load_weights
saving.load_weights_from_hdf5_group_by_name(f, layers)
File “/usr/lib/python3/dist-packages/keras/engine/topology.py”, line 3205, in load_weights_from_hdf5_group_by_name
K.batch_set_value(weight_value_tuples)
File “/usr/lib/python3/dist-packages/keras/backend/tensorflow_backend.py”, line 2247, in batch_set_value
get_session().run(assign_ops, feed_dict=feed_dict)
File “/usr/lib/python3/dist-packages/keras/backend/tensorflow_backend.py”, line 188, in get_session
session.run(tf.variables_initializer(uninitialized_vars))
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 950, in run
run_metadata_ptr)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1173, in _run
feed_dict_tensor, options, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1350, in _do_run
run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1024,324] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node mrcnn_bbox_fc/random_uniform/RandomUniform (defined at /usr/lib/python3/dist-packages/keras/backend/tensorflow_backend.py:3631) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Original stack trace for ‘mrcnn_bbox_fc/random_uniform/RandomUniform’:
File “”, line 1, in
File “/usr/lib/python3.6/idlelib/run.py”, line 144, in main
ret = method(*args, **kwargs)
File “/usr/lib/python3.6/idlelib/run.py”, line 474, in runcode
exec(code, self.locals)
File “/home/safran/Mask-RCNN-series-master/process_video.py”, line 3, in
from visualize_cv2 import model, display_instances, class_names
File “”, line 971, in _find_and_load
File “”, line 955, in _find_and_load_unlocked
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/home/safran/Mask-RCNN-series-master/visualize_cv2.py”, line 25, in
mode=“inference”, model_dir=MODEL_DIR, config=config
File “/home/safran/Mask-RCNN-series-master/model.py”, line 1837, in init
self.keras_model = self.build(mode=mode, config=config)
File “/home/safran/Mask-RCNN-series-master/model.py”, line 2038, in build
fc_layers_size=config.FPN_CLASSIF_FC_LAYERS_SIZE)
File “/home/safran/Mask-RCNN-series-master/model.py”, line 948, in fpn_classifier_graph
name=‘mrcnn_bbox_fc’)(shared)
File “/usr/lib/python3/dist-packages/keras/engine/topology.py”, line 576, in call
self.build(input_shapes[0])
File “/usr/lib/python3/dist-packages/keras/layers/wrappers.py”, line 154, in build
self.layer.build(child_input_shape)
File “/usr/lib/python3/dist-packages/keras/layers/core.py”, line 830, in build
constraint=self.kernel_constraint)
File “/usr/lib/python3/dist-packages/keras/legacy/interfaces.py”, line 87, in wrapper
return func(*args, **kwargs)
File “/usr/lib/python3/dist-packages/keras/engine/topology.py”, line 397, in add_weight
weight = K.variable(initializer(shape),
File “/usr/lib/python3/dist-packages/keras/initializers.py”, line 212, in call
dtype=dtype, seed=self.seed)
File “/usr/lib/python3/dist-packages/keras/backend/tensorflow_backend.py”, line 3631, in random_uniform
dtype=dtype, seed=seed)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/random_ops.py”, line 247, in random_uniform
rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_random_ops.py”, line 820, in random_uniform
name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py”, line 788, in _apply_op_helper
op_def=op_def)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py”, line 3616, in create_op
op_def=op_def)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py”, line 2005, in init
self._traceback = tf_stack.extract_stack()

I understand that this is a memory error, please guid me of what i have to do, I could not get any help online. If ur answere it to convert it using tensorrt then please guid me in the inference too

2

Hi,

This error is caused by out of memory.

ResourceExhaustedError: OOM when allocating tensor with shape ...

Here are two initial suggestions for you:

1. You can try to limit the maximal memory allocated by the TensorFlow.

from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.40 # 40% of memory
config.log_device_placement = True # to log device placement (on which device the operation ran)
sess = tf.Session(config=config)
set_session(sess) # set this TensorFlow session as the default session for Keras

However, if the model requires more memory than Nano maximum, you might still not able to get it work.

2. Convert the model into TensorRT.
We have a tutorial to share how to build a TensorRT engine from mask-RCNN model.
Please check this GitHub for the detail instructions:
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleUffMaskRCNN

Thanks.