Running a MaskRCNN Tensorflow saved model in Jetson Xavier - Gets killed

  1. We converted a Mask RCNN model to a TF Saved Model. Huge pb file - 255MB

  2. Tried to run it using Tensorflow-gpu on jetson xavier. version=1.13.1+nv19.3
    16GB RAM

  3. Using os.environ[“CUDA_VISIBLE_DEVICES”]=“0” it gets killed after some time

  4. I noticed that the 15+GB goes to 100% before it is killed

  5. Using os.environ[“CUDA_VISIBLE_DEVICES”]=“1” it runs fine but takes a long time (thats not what we want)

  6. It runs fine on a normal GPU desktop with GeForce GTK 1060 + 16GB + Corei7


Code below

import tensorflow as tf
import numpy as np

import os
os.environ[“CUDA_VISIBLE_DEVICES”]=“0”

export_path = “./serving_model/1”

config = tf.ConfigProto()
#config.gpu_options.per_process_gpu_memory_fraction = 0.4
config.log_device_placement=True
config.allow_soft_placement=True

used with tf.device(’/gpu:0’): and os.environ[“CUDA_VISIBLE_DEVICES”]=“0”

together and alternatively. no success

with tf.device(’/gpu:0’):
with tf.Session(config=config) as sess:
loaded = tf.saved_model.loader.load(sess, [“serve”], export_path)
graph = tf.get_default_graph()

sess.run(tf.global_variables_initializer())
print('session run initialized..')

opt = []

x_tensor1 = graph.get_tensor_by_name("input_image:0")
x_tensor2 = graph.get_tensor_by_name("input_image_meta:0")
x_tensor3 = graph.get_tensor_by_name("input_anchors:0")
op_to_restore = graph.get_tensor_by_name("mrcnn_detection/Reshape_1:0")

x_test1 = np.zeros(shape=(1,1024,1024,3), dtype=np.float)
x_test2 = np.zeros(shape=(1,14), dtype=np.float)
x_test3 = np.zeros(shape=(1,261888,4), dtype=np.float)
feed_dict = {x_tensor1: x_test1,x_tensor2: x_test2,x_tensor3: x_test3}

print('before loop')

import time
for i in range(1):
  t1 = time.time()
  opt = sess.run(op_to_restore, feed_dict)
  t2 = time.time()
  print('i = {} time taken: {}'.format(i, (t2 - t1)))
print(opt)

Forgot to add this:

Tensorflow-gpu version=1.13.1+nv19.3
CUDA Version 10.0.117
cuDNN 7.3.1

Adding this option did the trick

config.gpu_options.allow_growth=True

Hi shanky001, thanks for posting update with your resolution. Also see this post for the per_process_gpu_memory_fraction if that offers further help:

https://devtalk.nvidia.com/default/topic/1051522/jetson-nano/official-tensorflow-uses-all-ram-when-running-on-gpu/post/5337349/#5337349

Thanks.

Will try it along with the allow_growth option and see if it helps.