Convert SSD-Mobilenet to UFF

sivaishere96 · November 16, 2019, 2:57pm

I have a SSD-Mobilenetv2 trained on my own custom dataset using tensorflow. I plan on converting it to UFF to be used by deepstream. The conversion seem to run fine while using the convert-to-uff.py file while throwing a few warnings. However while running the tensorrt code it throws an error.

This is the output of convert-to-uff.py command

god@jetson-nano:/usr/lib/python2.7/dist-packages/uff/bin$ python3 convert_to_uff.py --input-file /home/god/frozen_inference_graph.pb -O NMS -p /usr/src/tensorrt/samples/sampleUffSSD/config.py -t
2019-11-16 20:24:15.936610: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /usr/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py:18: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

Loading /home/god/frozen_inference_graph.pb
WARNING:tensorflow:From /usr/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py:231: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

NOTE: UFF has been tested with TensorFlow 1.12.0. Other versions are not guaranteed to work
WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.
WARNING:tensorflow:From /usr/lib/python3.6/dist-packages/graphsurgeon/_utils.py:2: The name tf.NodeDef is deprecated. Please use tf.compat.v1.NodeDef instead.

WARNING: To create TensorRT plugin nodes, please use the `create_plugin_node` function instead.
WARNING: To create TensorRT plugin nodes, please use the `create_plugin_node` function instead.
UFF Version 0.6.3
=== Automatically deduced input nodes ===
[name: "Input"
op: "Placeholder"
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "shape"
  value {
    shape {
      dim {
        size: 1
      }
      dim {
        size: 3
      }
      dim {
        size: 300
      }
      dim {
        size: 300
      }
    }
  }
}
]
=========================================

Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS_TRT yet.
Converting NMS as custom op: NMS_TRT
WARNING:tensorflow:From /usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py:179: The name tf.AttrValue is deprecated. Please use tf.compat.v1.AttrValue instead.

Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_conf as custom op: FlattenConcat_TRT
Warning: No conversion function registered for layer: GridAnchor_TRT yet.
Converting GridAnchor as custom op: GridAnchor_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_loc as custom op: FlattenConcat_TRT
No. nodes: 661
UFF Output written to /home/god/frozen_inference_graph.uff
UFF Text Output written to /home/god/frozen_inference_graph.pbtxt

where config.py is defined as

import graphsurgeon as gs
import tensorflow as tf

Input = gs.create_node("Input",
    op="Placeholder",
    dtype=tf.float32,
    shape=[1, 3, 300, 300])
PriorBox = gs.create_plugin_node(name="GridAnchor", op="GridAnchor_TRT",
    numLayers=6,
    minSize=0.2,
    maxSize=0.95,
    aspectRatios=[1.0, 2.0, 0.5, 3.0, 0.33],
    variance=[0.1,0.1,0.2,0.2],
    featureMapShapes=[19, 10, 5, 3, 2, 1])
NMS = gs.create_plugin_node(name="NMS", op="NMS_TRT",
    shareLocation=1,
    varianceEncodedInTarget=0,
    backgroundLabelId=0,
    confidenceThreshold=1e-8,
    nmsThreshold=0.6,
    topK=100,
    keepTopK=100,
    numClasses=91,
    inputOrder=[0, 2, 1],
    confSigmoid=1,
    isNormalized=1)
concat_priorbox = gs.create_node(name="concat_priorbox", op="ConcatV2", dtype=tf.float32, axis=2)
concat_box_loc = gs.create_plugin_node("concat_box_loc", op="FlattenConcat_TRT", dtype=tf.float32, axis=1, ignoreBatch=0)
concat_box_conf = gs.create_plugin_node("concat_box_conf", op="FlattenConcat_TRT", dtype=tf.float32, axis=1, ignoreBatch=0)

namespace_plugin_map = {
    "MultipleGridAnchorGenerator": PriorBox,
    "Postprocessor": NMS,
    "Preprocessor": Input,
    "ToFloat": Input,
    "image_tensor": Input,
    "MultipleGridAnchorGenerator/Concatenate": concat_priorbox,
    "MultipleGridAnchorGenerator/Identity": concat_priorbox,
    "concat": concat_box_loc,
    "concat_1": concat_box_conf
}

def preprocess(dynamic_graph):
    # Now create a new graph by collapsing namespaces
    dynamic_graph.collapse_namespaces(namespace_plugin_map)
    # Remove the outputs, so we just have a single output node (NMS).
    dynamic_graph.remove(dynamic_graph.graph_outputs, remove_exclusive_dependencies=False)

This is the error while running the tensorrt code. I followed the official benchmarking instructions.

god@jetson-nano:/usr/src/tensorrt/bin$ sudo ./sample_uff_ssd_rect
../data/ssd/sample_unpruned_mobilenet_v2.uff
Registering UFF model
Registered Input
Registered output NMS
Creating engine
Begin parsing model...
[libprotobuf FATAL /home/erisuser/p4sw/sw/gpgpu/MachineLearning/DIT/externals/protobuf/aarch64/10.0/include/google/protobuf/repeated_field.h:1408] CHECK failed: (index) < (current_size_): 
terminate called after throwing an instance of 'google_private::protobuf::FatalException'
  what():  CHECK failed: (index) < (current_size_): 
Aborted

AastaLLL · November 18, 2019, 6:54am

Hi,

A similar issue is reported before and end up with the conflict of protobuf.
Could you share the protobuf version of your model and Nano with us first?

Thanks.

sivaishere96 · November 18, 2019, 8:13am

Both the versions are 3.10.0

AastaLLL · November 19, 2019, 3:04am

Hi,

We want to reproduce this issue in our environment.
Could you share following file with us?

/home/god/frozen_inference_graph.pb
/home/god/frozen_inference_graph.uff

Thanks.

sivaishere96 · November 19, 2019, 3:09am

Can you please share your email id. Thank you.

sivaishere96 · November 19, 2019, 8:18am

UPDATE: I tried running GitHub - AastaNV/TRT_object_detection: Python sample for referencing object detection model with TensorRT . This was the error I was getting:

[libprotobuf FATAL /home/erisuser/p4sw/sw/gpgpu/MachineLearning/DIT/externals/protobuf/aarch64/10.0/include/google/protobuf/repeated_field.h:1408] CHECK failed: (index) < (current_size_): 
Traceback (most recent call last):
  File "main.py", line 40, in <module>
    parser.parse('tmp.uff', network)
RuntimeError: CHECK failed: (index) < (current_size_):

AastaLLL · November 20, 2019, 3:07am

Hi,

Could you share the model via private message?
Thanks.

sivaishere96 · November 20, 2019, 3:09am

Yes I did. I sent you a google drive link 20 hours ago. Let me know if you need anything else.

sivaishere96 · November 22, 2019, 9:59am

Any updates AastaLLL?

AastaLLL · November 26, 2019, 7:53am

Hi,

Sorry that we still need some time to check this issue.
Will keep you updated.

Thanks.

sivaishere96 · November 29, 2019, 10:44am

Hi AAstaLLL,

It has been over three weeks and i have not yet got a solution to this problem.I have scoured through a lot of posts on this forum and it seems a lot of people have not been able to implement their custom SSD model. A lot of the threads have not been closed and are without solutions.

We buy jetson devices due to the SSD-Mobilenet-v2 benchmarks that have been advertised and in hopes that we could implement our model.

Not being able to implement custom SSD models defeats the whole purpose. There should be some official Nvidia blog for training SSD-Mobilenet or shoudl be supported in TLT. The lack of support for this problem makes me rethink the use of jetson devices in the long run for production purposes.

sivaishere96 · December 4, 2019, 7:22am

Hi AAstaLLL,

I have an update. I got the model working on deepstream with a few quirks. I will open another thread for that. Here are the steps I followed to get my custom trained single class SSD-Mobilenet-v2 working on deesptream.

Trained the SSD-Mobilenet-v2 model on a x86 server using tensorflow 1.12.xx
Export the model to .pb file.
Convert the forzen .pb file to uff using convert_to_uff.py on jetson.

god@jetson-nano:~$ sudo python3 /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py /home/god/Downloads/model_exported/frozen_inference_graph.pb -o ssd_mobilenet_v2.uff -O NMS -p /home/god/Downloads/config.py

The config file used,

import graphsurgeon as gs
import tensorflow as tf

Input = gs.create_node("Input",
    op="Placeholder",
    dtype=tf.float32,
    shape=[1, 3, 300, 300])
PriorBox = gs.create_plugin_node(name="GridAnchor", op="GridAnchor_TRT",
    numLayers=6,
    minSize=0.2,
    maxSize=0.95,
    aspectRatios=[1.0, 2.0, 0.5, 3.0, 0.33],
    variance=[0.1,0.1,0.2,0.2],
    featureMapShapes=[19, 10, 5, 3, 2, 1])
NMS = gs.create_plugin_node(name="NMS", op="NMS_TRT",
    shareLocation=1,
    varianceEncodedInTarget=0,
    backgroundLabelId=0,
    confidenceThreshold=1e-8,
    nmsThreshold=0.6,
    topK=100,
    keepTopK=100,
    numClasses=2,
    ###########################################
    inputOrder=[0, 2, 1],
    #inputOrder=[2, 0, 1],
    ###########################################
    confSigmoid=1,
    isNormalized=1,
    scoreConverter="SIGMOID")
concat_priorbox = gs.create_node(name="concat_priorbox", op="ConcatV2", dtype=tf.float32, axis=2)
concat_box_loc = gs.create_plugin_node("concat_box_loc", op="FlattenConcat_TRT", dtype=tf.float32, axis=1, ignoreBatch=0)
concat_box_conf = gs.create_plugin_node("concat_box_conf", op="FlattenConcat_TRT", dtype=tf.float32, axis=1, ignoreBatch=0)

namespace_plugin_map = {
    "MultipleGridAnchorGenerator": PriorBox,
    "Postprocessor": NMS,
    "Preprocessor": Input,
    "ToFloat": Input,
    "image_tensor": Input,
    #"MultipleGridAnchorGenerator/Concatenate": concat_priorbox,
    "Concatenate": concat_priorbox,
    "concat": concat_box_loc,
    "concat_1": concat_box_conf
}

def preprocess(dynamic_graph):
    all_assert_nodes = dynamic_graph.find_nodes_by_op("Assert")
    dynamic_graph.remove(all_assert_nodes, remove_exclusive_dependencies=True)

    all_identity_nodes = dynamic_graph.find_nodes_by_op("Identity")
    dynamic_graph.forward_inputs(all_identity_nodes)

    dynamic_graph.collapse_namespaces(namespace_plugin_map)
    dynamic_graph.remove(dynamic_graph.graph_outputs, remove_exclusive_dependencies=False)
    dynamic_graph.find_nodes_by_op("NMS_TRT")[0].input.remove("Input")

The important thing to note here is that if the number of classes is n then the number of classes in the config file should be n+1 and the inputOrder should be [0,2,1].

Use the .uff file in objectDetector_SSD in deepstream with the number of classes being n+1
and the output-blob-names being “NMS”. Also change the number of classes in
nvdsparsebbox_ssd.cpp to n+1. Add “background” class to your labels.txt else you
would not see any output detections.

Thanks for all your help AastaLLL. However the model is not yet working as expected. I am posting the problem in an another thread.

https://devtalk.nvidia.com/default/topic/1067688/deepstream-sdk/custom-trained-ssd-mobilenet-v2-output-detections-are-jittery-/

leandrovrabelo · May 26, 2020, 12:08pm

Hi, I was with a similar problem and the following tutorial helped me a lot:

Topic		Replies	Views
Problems with SSD Mobilenet v2 UFF Jetson Nano ssd	35	7937	October 18, 2021
how to write config.py for converting ssd-mobilenetv2 to uff format Jetson Nano	19	6879	October 14, 2021
How to convert SSD mobilenet v2 to uff,Then use uff in jetson_inference detectnet_camera script？ Jetson Nano	13	2172	October 14, 2021
Problem with SSD Mobilenet TensorRT conversion Jetson Nano tensorrt	7	966	October 15, 2021
How adapt Tensorflow object detection for custom dataset to Deepstream 5.0 DeepStream SDK tensorflow	17	1978	July 27, 2021
Convert To Uff failing on Object Detection API Tf1 model Jetson Nano tensorrt	5	1745	October 15, 2021
Cannot convert a tensorflow model to UFF Jetson TX2	13	2335	October 18, 2021
How to retrain ssd_inception_v2_coco_2017_11_17 from the tensorrt - samples Jetson TX2	33	6947	October 18, 2021
Getting up to the Jetson inference performance Jetson AGX Xavier tensorrt , jetson-inference	19	964	October 18, 2021
sampleUffSSD with custom ssd_mobilenet_v1 model TensorRT	37	4490	October 12, 2021

Convert SSD-Mobilenet to UFF

Related topics