MTCNN tensorflow into tensorflow tensorRT on Jetson Nano

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

I am trying to optimise tensorflow implementation of the MTCNN https://github.com/AITTSMD/MTCNN-Tensorflow

I used

output_node_names = [n.name for n in tf.get_default_graph().as_graph_def().node]
print(output_node_names)

After line 37 https://github.com/AITTSMD/MTCNN-Tensorflow/blob/master/Detection/fcn_detector.py?#L37 in fcn_detector.py to know the node names. I got this output

['input_image', 'image_width', 'image_height', 'Reshape/shape/0', 'Reshape/shape/3', 'Reshape/shape', 'Reshape', 'conv1/weights/Initializer/random_uniform/shape', 'conv1/weights/Initializer/random_uniform/min', 'conv1/weights/Initializer/random_uniform/max', 'conv1/weights/Initializer/random_uniform/RandomUniform', 'conv1/weights/Initializer/random_uniform/sub', 'conv1/weights/Initializer/random_uniform/mul', 'conv1/weights/Initializer/random_uniform', 'conv1/weights', 'conv1/weights/Assign', 'conv1/weights/read', 'conv1/kernel/Regularizer/l2_regularizer/scale', 'conv1/kernel/Regularizer/l2_regularizer/L2Loss', 'conv1/kernel/Regularizer/l2_regularizer', 'conv1/biases/Initializer/zeros', 'conv1/biases', 'conv1/biases/Assign', 'conv1/biases/read', 'conv1/dilation_rate', 'conv1/Conv2D', 'conv1/BiasAdd', 'conv1/alphas/Initializer/Const', 'conv1/alphas', 'conv1/alphas/Assign', 'conv1/alphas/read', 'conv1/Relu', 'conv1/Abs', 'conv1/sub', 'conv1/mul', 'conv1/mul_1/y', 'conv1/mul_1', 'conv1/add', 'conv1/add/activations/tag', 'conv1/add/activations', 'pool1/MaxPool', 'pool1/MaxPool/activations/tag', 'pool1/MaxPool/activations', 'conv2/weights/Initializer/random_uniform/shape', 'conv2/weights/Initializer/random_uniform/min', 'conv2/weights/Initializer/random_uniform/max', 'conv2/weights/Initializer/random_uniform/RandomUniform', 'conv2/weights/Initializer/random_uniform/sub', 'conv2/weights/Initializer/random_uniform/mul', 'conv2/weights/Initializer/random_uniform', 'conv2/weights', 'conv2/weights/Assign', 'conv2/weights/read', 'conv2/kernel/Regularizer/l2_regularizer/scale', 'conv2/kernel/Regularizer/l2_regularizer/L2Loss', 'conv2/kernel/Regularizer/l2_regularizer', 'conv2/biases/Initializer/zeros', 'conv2/biases', 'conv2/biases/Assign', 'conv2/biases/read', 'conv2/dilation_rate', 'conv2/Conv2D', 'conv2/BiasAdd', 'conv2/alphas/Initializer/Const', 'conv2/alphas', 'conv2/alphas/Assign', 'conv2/alphas/read', 'conv2/Relu', 'conv2/Abs', 'conv2/sub', 'conv2/mul', 'conv2/mul_1/y', 'conv2/mul_1', 'conv2/add', 'conv2/add/activations/tag', 'conv2/add/activations', 'conv3/weights/Initializer/random_uniform/shape', 'conv3/weights/Initializer/random_uniform/min', 'conv3/weights/Initializer/random_uniform/max', 'conv3/weights/Initializer/random_uniform/RandomUniform', 'conv3/weights/Initializer/random_uniform/sub', 'conv3/weights/Initializer/random_uniform/mul', 'conv3/weights/Initializer/random_uniform', 'conv3/weights', 'conv3/weights/Assign', 'conv3/weights/read', 'conv3/kernel/Regularizer/l2_regularizer/scale', 'conv3/kernel/Regularizer/l2_regularizer/L2Loss', 'conv3/kernel/Regularizer/l2_regularizer', 'conv3/biases/Initializer/zeros', 'conv3/biases', 'conv3/biases/Assign', 'conv3/biases/read', 'conv3/dilation_rate', 'conv3/Conv2D', 'conv3/BiasAdd', 'conv3/alphas/Initializer/Const', 'conv3/alphas', 'conv3/alphas/Assign', 'conv3/alphas/read', 'conv3/Relu', 'conv3/Abs', 'conv3/sub', 'conv3/mul', 'conv3/mul_1/y', 'conv3/mul_1', 'conv3/add', 'conv3/add/activations/tag', 'conv3/add/activations', 'conv4_1/weights/Initializer/random_uniform/shape', 'conv4_1/weights/Initializer/random_uniform/min', 'conv4_1/weights/Initializer/random_uniform/max', 'conv4_1/weights/Initializer/random_uniform/RandomUniform', 'conv4_1/weights/Initializer/random_uniform/sub', 'conv4_1/weights/Initializer/random_uniform/mul', 'conv4_1/weights/Initializer/random_uniform', 'conv4_1/weights', 'conv4_1/weights/Assign', 'conv4_1/weights/read', 'conv4_1/kernel/Regularizer/l2_regularizer/scale', 'conv4_1/kernel/Regularizer/l2_regularizer/L2Loss', 'conv4_1/kernel/Regularizer/l2_regularizer', 'conv4_1/biases/Initializer/zeros', 'conv4_1/biases', 'conv4_1/biases/Assign', 'conv4_1/biases/read', 'conv4_1/dilation_rate', 'conv4_1/Conv2D', 'conv4_1/BiasAdd', 'conv4_1/Softmax', 'conv4_1/Softmax/activations/tag', 'conv4_1/Softmax/activations', 'conv4_2/weights/Initializer/random_uniform/shape', 'conv4_2/weights/Initializer/random_uniform/min', 'conv4_2/weights/Initializer/random_uniform/max', 'conv4_2/weights/Initializer/random_uniform/RandomUniform', 'conv4_2/weights/Initializer/random_uniform/sub', 'conv4_2/weights/Initializer/random_uniform/mul', 'conv4_2/weights/Initializer/random_uniform', 'conv4_2/weights', 'conv4_2/weights/Assign', 'conv4_2/weights/read', 'conv4_2/kernel/Regularizer/l2_regularizer/scale', 'conv4_2/kernel/Regularizer/l2_regularizer/L2Loss', 'conv4_2/kernel/Regularizer/l2_regularizer', 'conv4_2/biases/Initializer/zeros', 'conv4_2/biases', 'conv4_2/biases/Assign', 'conv4_2/biases/read', 'conv4_2/dilation_rate', 'conv4_2/Conv2D', 'conv4_2/BiasAdd', 'conv4_2/BiasAdd/activations/tag', 'conv4_2/BiasAdd/activations', 'conv4_3/weights/Initializer/random_uniform/shape', 'conv4_3/weights/Initializer/random_uniform/min', 'conv4_3/weights/Initializer/random_uniform/max', 'conv4_3/weights/Initializer/random_uniform/RandomUniform', 'conv4_3/weights/Initializer/random_uniform/sub', 'conv4_3/weights/Initializer/random_uniform/mul', 'conv4_3/weights/Initializer/random_uniform', 'conv4_3/weights', 'conv4_3/weights/Assign', 'conv4_3/weights/read', 'conv4_3/kernel/Regularizer/l2_regularizer/scale', 'conv4_3/kernel/Regularizer/l2_regularizer/L2Loss', 'conv4_3/kernel/Regularizer/l2_regularizer', 'conv4_3/biases/Initializer/zeros', 'conv4_3/biases', 'conv4_3/biases/Assign', 'conv4_3/biases/read', 'conv4_3/dilation_rate', 'conv4_3/Conv2D', 'conv4_3/BiasAdd', 'conv4_3/BiasAdd/activations/tag', 'conv4_3/BiasAdd/activations', 'Squeeze', 'Squeeze_1', 'Squeeze_2', 'save/filename/input', 'save/filename', 'save/Const', 'save/SaveV2/tensor_names', 'save/SaveV2/shape_and_slices', 'save/SaveV2', 'save/control_dependency', 'save/RestoreV2/tensor_names', 'save/RestoreV2/shape_and_slices', 'save/RestoreV2', 'save/Assign', 'save/Assign_1', 'save/Assign_2', 'save/Assign_3', 'save/Assign_4', 'save/Assign_5', 'save/Assign_6', 'save/Assign_7', 'save/Assign_8', 'save/Assign_9', 'save/Assign_10', 'save/Assign_11', 'save/Assign_12', 'save/Assign_13', 'save/Assign_14', 'save/restore_all']

for PNET

I did similar thing for ONET and RNET after line 25 https://github.com/AITTSMD/MTCNN-Tensorflow/blob/master/Detection/detector.py?#L25 in detector.py

I got the same names for the input and output nodes for these as well.

So I used save/restore_all for convert pb for PNET

output_node_name = ['save/restore_all']
# Freeze the graph
            frozen_graph_def = tf.graph_util.convert_variables_to_constants(self.sess,self.sess.graph_def,output_node_name) 
            with open('p_output_graph.pb', 'wb') as f:
                f.write(frozen_graph_def.SerializeToString())

after line 37 https://github.com/AITTSMD/MTCNN-Tensorflow/blob/master/Detection/fcn_detector.py?#L37 in fcn_detector.py

Similar for ONET and RNET after line 25 after line 25 https://github.com/AITTSMD/MTCNN-Tensorflow/blob/master/Detection/detector.py?#L25 in detector.py

Convert all the pb to trt pb

import tensorflow as tf

import tensorflow.contrib.tensorrt as trt

# First deserialize your frozen graph:
with tf.gfile.GFile('/home/jetsonnano/Downloads/MTCNN-Tensorflow-master/test/p_output_graph.pb', 'rb') as f:
	frozen_graph = tf.GraphDef()
	frozen_graph.ParseFromString(f.read())
	for node in frozen_graph.node:
		if node.op == 'RefSwitch':
			node.op = 'Switch'
			for index in xrange(len(node.input)):
				if 'moving_' in node.input[index]:
					node.input[index] = node.input[index] + '/read'
		elif node.op == 'AssignSub':
			node.op = 'Sub'
			if 'use_locking' in node.attr: del node.attr['use_locking']
		elif node.op == 'AssignAdd':
			node.op = 'Add'
			if 'use_locking' in node.attr: del node.attr['use_locking']
		elif node.op == 'Assign':
			node.op = 'Identity'
			if 'use_locking' in node.attr: del node.attr['use_locking']
			if 'validate_shape' in node.attr: del node.attr['validate_shape']
			if len(node.input) == 2:
				# input0: ref: Should be from a Variable node. May be uninitialized.
				# input1: value: The value to be assigned to the variable.
				node.input[0] = node.input[1]
				del node.input[1]
# Now you can create a TensorRT inference graph from your
#save/restore_all
# frozen graph:
	converter = trt.create_inference_graph(
	input_graph_def=frozen_graph,
	precision_mode='FP16',
	outputs=['save/restore_all']) #output nodes
# ~ trt_graph = converter.convert()

	tf.train.write_graph(converter, "model", "p_output_graph_FP16.pb", as_text=False)

Similar for ONET and RNET

I changed from Line 7 to Line 37 in https://github.com/AITTSMD/MTCNN-Tensorflow/blob/master/Detection/fcn_detector.py?#L7 in fcn_detector.py

class FcnDetector(object):
    #net_factory: which net
    #model_path: where the params'file is
    def __init__(self, net_factory, model_path):
        #create a graph
        graph = tf.Graph()
        with graph.as_default():
            #define tensor and op in graph(-1,1)
            self.image_op = tf.placeholder(tf.float32, name='input_image')
            self.width_op = tf.placeholder(tf.int32, name='image_width')
            self.height_op = tf.placeholder(tf.int32, name='image_height')
            image_reshape = tf.reshape(self.image_op, [1, self.height_op, self.width_op, 3])
            #self.cls_prob batch*2
            #self.bbox_pred batch*4
            #construct model here
            #self.cls_prob, self.bbox_pred = net_factory(image_reshape, training=False)
            #contains landmark
            self.cls_prob, self.bbox_pred, _ = net_factory(image_reshape, training=False)
            
            #allow 
            self.sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, gpu_options=tf.GPUOptions(allow_growth=True)))
            saver = tf.train.Saver()
            #check whether the dictionary is valid
            model_dict = '/'.join(model_path.split('/')[:-1])
            ckpt = tf.train.get_checkpoint_state(model_dict)
            print(model_path)
            readstate = ckpt and ckpt.model_checkpoint_path
            assert  readstate, "the params dictionary is not valid"
            print("restore models' param")
            saver.restore(self.sess, model_path)
            output_node_names = [n.name for n in tf.get_default_graph().as_graph_def().node]

to

class FcnDetector(object):
    #net_factory: which net
    #model_path: where the params'file is
    def __init__(self, net_factory, model_path):
        #create a graph
        graph = tf.GraphDef()
        
        # Import the TF graph
        with tf.gfile.GFile(model_path, 'rb') as f:
            graph.ParseFromString(f.read())
            tf.import_graph_def(graph, name='')

        self.image_op = tf.placeholder(tf.float32, name='input_image')
        self.width_op = tf.placeholder(tf.int32, name='image_width')
        self.height_op = tf.placeholder(tf.int32, name='image_height')
        image_reshape = tf.reshape(self.image_op, [1, self.height_op, self.width_op, 3])
        self.cls_prob, self.bbox_pred, _ = net_factory(image_reshape, training=False)
		
		#allow 
        self.sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, gpu_options=tf.GPUOptions(allow_growth=True)))

I changed from Line 6 to Line 25 in https://github.com/AITTSMD/MTCNN-Tensorflow/blob/master/Detection/detector.py?#L6 in detector.py

from

class Detector(object):
    #net_factory:rnet or onet
    #datasize:24 or 48
    def __init__(self, net_factory, data_size, batch_size, model_path):
        graph = tf.Graph()
        with graph.as_default():
            self.image_op = tf.placeholder(tf.float32, shape=[batch_size, data_size, data_size, 3], name='input_image')
            #figure out landmark            
            self.cls_prob, self.bbox_pred, self.landmark_pred = net_factory(self.image_op, training=False)
            self.sess = tf.Session(
                config=tf.ConfigProto(allow_soft_placement=True, gpu_options=tf.GPUOptions(allow_growth=True)))
            saver = tf.train.Saver()
            #check whether the dictionary is valid
            model_dict = '/'.join(model_path.split('/')[:-1])
            ckpt = tf.train.get_checkpoint_state(model_dict)
            print(model_path)
            readstate = ckpt and ckpt.model_checkpoint_path
            assert  readstate, "the params dictionary is not valid"
            print("restore models' param")
            saver.restore(self.sess, model_path)

to

class Detector(object):
    #net_factory:rnet or onet
    #datasize:24 or 48
    def __init__(self, net_factory, data_size, batch_size, model_path):
        graph = tf.GraphDef()
        
        # Import the TF graph
        with tf.gfile.GFile(model_path, 'rb') as f:
            graph.ParseFromString(f.read())
            tf.import_graph_def(graph, name='')        
        
        # ~ with graph():
        self.image_op = tf.placeholder(tf.float32, shape=[batch_size, data_size, data_size, 3], name='input_image')
		#figure out landmark            
        self.cls_prob, self.bbox_pred, self.landmark_pred = net_factory(self.image_op, training=False)
        self.sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, gpu_options=tf.GPUOptions(allow_growth=True)))

Then run camera_test.py https://github.com/AITTSMD/MTCNN-Tensorflow/blob/master/test/camera_test.py
after changing to the new path pointing to the pd file of PNET,ONET and RNET

I get this error

Traceback (most recent call last):
  File "camera_test_trrt.py", line 26, in <module>
    RNet = Detector(R_Net, 24, 1, '/home/jetsonnano/Downloads/MTCNN-Tensorflow-master/test/r_output_graph_FP16.pb')
  File "../Detection/detector_trrt.py", line 20, in __init__
    self.cls_prob, self.bbox_pred, self.landmark_pred = net_factory(self.image_op, training=False)
  File "../train_models/mtcnn_model.py", line 257, in R_Net
    net = slim.conv2d(inputs, num_outputs=28, kernel_size=[3,3], stride=1, scope="conv1")
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1158, in convolution2d
    conv_dims=2)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1061, in convolution
    outputs = layer.apply(inputs)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1227, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 530, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 538, in __call__
    self._maybe_build(inputs)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1603, in _maybe_build
    self.build(input_shapes)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/keras/layers/convolutional.py", line 165, in build
    dtype=self.dtype)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 435, in add_weight
    getter=vs.get_variable)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 349, in add_weight
    aggregation=aggregation)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/training/checkpointable/base.py", line 607, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1479, in get_variable
    aggregation=aggregation)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1220, in get_variable
    aggregation=aggregation)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 530, in get_variable
    return custom_getter(**custom_getter_kwargs)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1753, in layer_variable_getter
    return _model_variable_getter(getter, *args, **kwargs)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1744, in _model_variable_getter
    aggregation=aggregation)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 350, in model_variable
    aggregation=aggregation)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 277, in variable
    aggregation=aggregation)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 499, in _true_getter
    aggregation=aggregation)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 848, in _get_single_variable
    traceback.format_list(tb))))
ValueError: Variable conv1/weights already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 277, in variable
    aggregation=aggregation)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/home/jetsonnano/.virtualenvs/jetsonnanotest/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/variables.py", line 350, in model_variable
    aggregation=aggregation)

What is going wrong?
I am confused as to if the same INPUT and OUTPUT name of PNET,ONET and RNET are causing this issue or is it something else.

2

I don’t know how to solve your problem. But I’ve written blog posts about how to optimize caffe MTCNN face detector with TensorRT and do inferencing on Jetson Nano before. Check it out if that interests you.

“TensorRT MTCNN Face Detector”: https://jkjung-avt.github.io/tensorrt-mtcnn/
“Optimizing TensorRT MTCNN”: https://jkjung-avt.github.io/optimize-mtcnn/

960×540 917 KB