Problems with SSD Mobilenet v2 UFF

Hi there,

i try to get my custom trained SSD Mobilenetv2 to work on my jetson nano with 1 class.
I manage to convert it to uff by using /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py
with this config.py

import graphsurgeon as gs
import tensorflow as tf

Input = gs.create_node("Input",
    op="Placeholder",
    dtype=tf.float32,
    shape=[1, 3, 300, 300])
PriorBox = gs.create_plugin_node(name="GridAnchor", op="GridAnchor_TRT",
    numLayers=6,
    minSize=0.2,
    maxSize=0.95,
    aspectRatios=[1.0, 2.0, 0.5, 3.0, 0.33],
    variance=[0.1,0.1,0.2,0.2],
    featureMapShapes=[19, 10, 5, 3, 2, 1])
NMS = gs.create_plugin_node(name="NMS", op="NMS_TRT",
    shareLocation=1,
    varianceEncodedInTarget=0,
    backgroundLabelId=0,
    confidenceThreshold=1e-8,
    nmsThreshold=0.6,
    topK=100,
    keepTopK=100,
    numClasses=1,
    inputOrder=[1, 0,2], #021 120 102(this seems to be the right one for me) 012 210
    confSigmoid=1,
    isNormalized=1)
concat_priorbox = gs.create_node(name="concat_priorbox", op="ConcatV2", dtype=tf.float32, axis=2)
concat_box_loc = gs.create_plugin_node("concat_box_loc", op="FlattenConcat_TRT", dtype=tf.float32, axis=1, ignoreBatch=0)
concat_box_conf = gs.create_plugin_node("concat_box_conf", op="FlattenConcat_TRT", dtype=tf.float32, axis=1, ignoreBatch=0)

namespace_plugin_map = {
    "MultipleGridAnchorGenerator": PriorBox,
    "Postprocessor": NMS,
    "Preprocessor": Input,
    "ToFloat": Input,
    "image_tensor": Input,
    "Concatenate": concat_priorbox,
    "concat": concat_box_loc,
    "concat_1": concat_box_conf
}

def preprocess(dynamic_graph):
    # Now create a new graph by collapsing namespaces
    dynamic_graph.collapse_namespaces(namespace_plugin_map)
    # Remove the outputs, so we just have a single output node (NMS).
    dynamic_graph.remove(dynamic_graph.graph_outputs, remove_exclusive_dependencies=False)
    # Disconnect the Input node from NMS, as it expects to have only 3 inputs.
    dynamic_graph.find_nodes_by_op("NMS_TRT")[0].input.remove("Input")

getting this warning in the terminal:

Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS_TRT yet.
Converting NMS as custom op: NMS_TRT
Warning: No conversion function registered for layer: GridAnchor_TRT yet.
Converting GridAnchor as custom op: GridAnchor_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_loc as custom op: FlattenConcat_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_conf as custom op: FlattenConcat_TRT
No. nodes: 1094

and getting this output as .pbtxt with the -t flag:

version: 1
descriptor_core_version: 1
descriptors {
  id: "tensorflow_extension"
  version: 1
}
descriptors {
  id: "custom"
  version: 1
}
graphs {
  id: "main"
  nodes {
    id: "NMS"
    inputs: "concat_box_conf"
    inputs: "Squeeze"
    inputs: "concat_priorbox"
    operation: "_NMS_TRT"
    fields {
      key: "backgroundLabelId_u_int"
      value {
        i: 0
      }
    }
    fields {
      key: "confSigmoid_u_int"
      value {
        i: 1
      }
    }
    fields {
      key: "confidenceThreshold_u_float"
      value {
        d: 1e-08
      }
    }
    fields {
      key: "inputOrder_u_ilist"
      value {
        i_list {
          val: 1
          val: 0
          val: 2
        }
      }
    }
    fields {
      key: "isNormalized_u_int"
      value {
        i: 1
      }
    }
    fields {
      key: "keepTopK_u_int"
      value {
        i: 100
      }
    }
    fields {
      key: "nmsThreshold_u_float"
      value {
        d: 0.6
      }
    }
    fields {
      key: "numClasses_u_int"
      value {
        i: 1
      }
    }
    fields {
      key: "shareLocation_u_int"
      value {
        i: 1
      }
    }
    fields {
      key: "topK_u_int"
      value {
        i: 100
      }
    }
    fields {
      key: "varianceEncodedInTarget_u_int"
      value {
        i: 0
      }
    }
  }
 
    }
    fields {
      key: "shape"
      value {
        i_list {
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_FeatureExtractor/MobilenetV2/Conv_1/BatchNorm/batchnorm/add/y"
      }
    }
  }

then I copied /usr/src/tensorrt/samples/python/uff_ssd and changed
numClasses=1,
inputOrder=[1, 0, 2],
in utils/model.py,

the Class List to only one item in utils/coco.py
and the ssd_model_uff_path to my newly created uff model in detect_objets.py.

when I run python3 detect_objects.py …

I get the following error

TensorRT inference engine settings:
  * Inference precision - DataType.FLOAT
  * Max batch size - 1

Building TensorRT engine. This may take few minutes.
python3: nmsPlugin.cpp:136: virtual void nvinfer1::plugin::DetectionOutput::configureWithFormat(const nvinfer1::Dims*, int, const nvinfer1::Dims*, int, nvinfer1::DataType, nvinfer1::PluginFormat, int): Assertion `numPriors * param.numClasses == inputDims[param.inputOrder[1]].d[0]' failed.
Aborted (core dumped)

I saw on other posts, that some had error 135 when they didn’t change the input order, but no post with error in line 136 of nmsPlugin.cpp.
Did I forget to do something special, to get it working?

here is my .pb and my .uff if this helps
https://drive.google.com/open?id=19dKoq7Jo_yRtF6LATE9DhF4XfF8ych9k

Hi,

1.

Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS_TRT yet.

This is a harmless warning since UFF converter cannot know if there’s a plugin for a given op.
You can just skip it.

2.

python3: nmsPlugin.cpp:136: virtual void nvinfer1::plugin::DetectionOutput::configureWithFormat(const nvinfer1::Dims*, int, const nvinfer1::Dims*, int, nvinfer1::DataType, nvinfer1::PluginFormat, int): Assertion `numPriors * param.numClasses == inputDims[param.inputOrder[1]].d[0]' failed.
Aborted (core dumped)

Error occurs on the case numPriors * param.numClasses != inputDims[param.inputOrder[1]].d[0].

I saw you change the inputOrder to [1,0,2]. This may change the dimension of axis=0 and leads to the error.
Could you try to set inputOrder=[0, 2, 1] first?

Thanks.

Hi @AastaLLL,
when I change the inputOrder to [0, 2, 1] (in the config.py and model.py) i get this Error

Building TensorRT engine. This may take few minutes.
python3: nmsPlugin.cpp:135: virtual void nvinfer1::plugin::DetectionOutput::configureWithFormat(const nvinfer1::Dims*, int, const nvinfer1::Dims*, int, nvinfer1::DataType, nvinfer1::PluginFormat, int): Assertion `numPriors * numLocClasses * 4 == inputDims[param.inputOrder[0]].d[0]' failed.
Aborted (core dumped)

this was my error that brought me to https://devtalk.nvidia.com/default/topic/1050465/jetson-nano/how-to-write-config-py-for-converting-ssd-mobilenetv2-to-uff-format/post/5331289/#5331289 in the first place.

Any Idea what the Error
nmsPlugin.cpp:136: virtual void nvinfer1::plugin::DetectionOutput::configureWithFormat(const nvinfer1::Dims*, int, const nvinfer1::Dims*, int, nvinfer1::DataType, nvinfer1::PluginFormat, int): Assertion `numPriors * param.numClasses == inputDims[param.inputOrder[1]].d[0]’ failed.

Could mean @AastaLLL ?

Hi,

The error occurs in the dimension incompatible.

The nmsPlugin is our internal plugin implementation and is not able to handle difference input dimension.
Since the output class has customized into 1, is there any change in the nms input/output dimension?

Thanks.

Hi AastaLLL,

I don’t really understand your question, youd you specify?

having problems while converting custom SSD Models to uff and then building an engine seems to be widely spread problem. Why is there no step by step documentation on how to do it?

Thanks

Hi,

You can find steps by steps document here:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#uffssd_sample

In short, there are some non-supported layers inside the SSD model.
This requires you the plugin implementation to make it work.

The SSD sample target for demonstrating how to inference a model with a plugin layer.
But the plugin layer is not flexible enough to handle different variance.
(Please understand this. It is not a standard layer but just a sample for showing plugin utility)

So you will need to implement your plugin layer for a customized SSD.
Thanks.

Hi,

I am having the same problem and getting the same error as above. I am on a TX2, using the exact same model as the sample (ssd_inception_v2_coco_2017_11_17) but with a different number of classes to predict.

Can you release the source code that was used to create the NMSplugin? It really doesn’t seem like a viable product release if you cannot customize even the number of prediction classes! Why go through the trouble to document things this far, if there are no real guides to modify or make custom implementations?

Thank you for your help.

Any one can help me ???
I also have the same problem to convert "“ssdlite_mobilenet_v2_coco_2018_05_09” model

/home/york/anaconda3/envs/tf1.13-gpu/bin/python3.5 /media/york/F/GitHub/tensorflow/models/research/uff_ssd_int8_python/trt_detect_objects.py
WARNING: To create TensorRT plugin nodes, please use the create_plugin_node function instead.
NOTE: UFF has been tested with TensorFlow 1.12.0. Other versions are not guaranteed to work
WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.
UFF Version 0.6.3
=== Automatically deduced input nodes ===
[name: “Input”
op: “Placeholder”
attr {
key: “dtype”
value {
type: DT_FLOAT
}
}
attr {
key: “shape”
value {
shape {
dim {
size: 1
}
dim {
size: 3
}
dim {
size: 300
}
dim {
size: 300
}
}
}
}
]

Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS_TRT yet.
Converting NMS as custom op: NMS_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_conf as custom op: FlattenConcat_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_loc as custom op: FlattenConcat_TRT
No. nodes: 589
UFF Output written to /media/york/F/GitHub/tensorflow/train_model/ssdlite_mobilenet_v2_coco_2018_05_09/frozen_inference_graph.uff
UFF Text Output written to /media/york/F/GitHub/tensorflow/train_model/ssdlite_mobilenet_v2_coco_2018_05_09/frozen_inference_graph.pbtxt
TensorRT inference engine settings:

  • Inference precision - DataType.FLOAT
  • Max batch size - 1

Building TensorRT engine. This may take few minutes.
python3.5: nmsPlugin.cpp:139: virtual void nvinfer1::plugin::DetectionOutput::configureWithFormat(const nvinfer1::Dims*, int, const nvinfer1::Dims*, int, nvinfer1::DataType, nvinfer1::PluginFormat, int): Assertion `numPriors * numLocClasses * 4 == inputDims[param.inputOrder[0]].d[0]’ failed.

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Hi,

NMS is not a open source plugin.
If your use case cannot be covered, it’s recommended to write your plugin directly.

Here is a sample for your reference:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#uff_custom_plugin

Thanks.

Hi,
With the below plugin for NMS, I could create the UFF for ssd InceptionV2. However it is failing while trying to create the trt engine. belo is the config.py

TRT Version : 5.1.2.2

import graphsurgeon as gs
import tensorflow as tf

Input = gs.create_node(“Input”,
op=“Placeholder”,
dtype=tf.float32,
shape=[1, 3, 300, 300])
PriorBox = gs.create_plugin_node(name=“GridAnchor”, op=“GridAnchor_TRT”,
numLayers=6,
minSize=0.2,
maxSize=0.95,
aspectRatios=[1.0, 2.0, 0.5, 3.0, 0.33],
variance=[0.1,0.1,0.2,0.2],
featureMapShapes=[19, 10, 5, 3, 2, 1])
#featureMapShapes=[38, 19, 10, 5, 3, 2])
NMS = gs.create_plugin_node(name=“NMS”, op=“NMS_TRT”,
shareLocation=1,
varianceEncodedInTarget=0,
backgroundLabelId=0,
confidenceThreshold=1e-8,
nmsThreshold=0.6,
topK=100,
keepTopK=100,
numClasses=7,
inputOrder=[0, 2, 1],
confSigmoid=1,
isNormalized=1,
scoreConverter=“SIGMOID”)

concat_priorbox = gs.create_node(name=“concat_priorbox”, op=“ConcatV2”, dtype=tf.float32, axis=2)
concat_box_loc = gs.create_node(“concat_box_loc”, op=“FlattenConcat_TRT”, dtype=tf.float32, axis=1, ignoreBatch=0)
concat_box_conf = gs.create_node(“concat_box_conf”, op=“FlattenConcat_TRT”, dtype=tf.float32, axis=1, ignoreBatch=0)

namespace_plugin_map = {
“MultipleGridAnchorGenerator”: PriorBox,
“Postprocessor”: NMS,
“Preprocessor”: Input,
# “ToFloat”: Input,
# “image_tensor”: Input,
#“MultipleGridAnchorGenerator/Concatenate”: concat_priorbox,
“Concatenate/concat”: concat_priorbox,
“concat”: concat_box_loc,
“concat_1”: concat_box_conf,
}

namespace_remove = {
“ToFloat”,
“image_tensor”,
“Preprocessor/map/TensorArrayStack_1/TensorArrayGatherV3”,
}

def preprocess(dynamic_graph):
print(’>>>>>>>>>>>> Inside preprocess ')
# remove the unrelated or error layers
dynamic_graph.remove(dynamic_graph.find_nodes_by_path(namespace_remove), remove_exclusive_dependencies=False)

# Now create a new graph by collapsing namespaces
dynamic_graph.collapse_namespaces(namespace_plugin_map)
# Remove the outputs, so we just have a single output node (NMS).
dynamic_graph.remove(dynamic_graph.graph_outputs, remove_exclusive_dependencies=False)

# Remove the Squeeze to avoid "Assertion `isPlugin(layerName)' failed"
Squeeze = dynamic_graph.find_node_inputs_by_name(dynamic_graph.graph_outputs[0], 'Squeeze')
dynamic_graph.forward_inputs(Squeeze)

Hi, did you solve your problem? I’m observing similar issues when trying to convert a custom graph with fewer classes than the original example. I can’t find anything useful on this topic, and the documentation is pretty bad.

Try replacing numClasses=1 with numClasses=2 , if you have trained for a single class model.

Hi, ashispapu

Would you mind to share the error when creating TRT engine?

uff parser targets for converting the network from .pb file into .uff.
It won’t valid the layer detail like dimension.

Thanks.

[TensorRT] ERROR: UffParser: Validator error: concat_box_loc: Unsupported operation _FlattenConcat_TRT

Hi,

Sorry if I didn’t mention this.

There are one plugin required by the ssd_mobilenet called FlattenConcat.
You can find it in our TensorRT sample: /usr/src/tensorrt/samples/python/uff_ssd

You can build it with the following command:

mkdir build
cd build
cmake ..
make

You will have the libflattenconcat.so library inside the build folder.
Then add the path into your python script:

ctypes.CDLL("/path/to/the/libflattenconcat.so")

Thanks.

Yes, I solved it by debugging. Thanks for the input.

Good to know this.

Hi @Tonto500

  1. On which Tensorflow version you have trained SSD Mobilenet V2 on custom data-sets?

  2. What is the hash value of your Tensorflow models repo on which you have generated inference graph?

Can you please check ?

I am getting error of unsupported _Cast operation.