Problem converting ONNX model to TensorRT Engine for SSD Mobilenet V2

Hi @god_ra,
I also used opset 11.

The solution does not fix everything, just the unsupported-UINT8 error.
So if there is still another problem while parsing the uff-file this will not be fixed by only modifying the input node.

I used opset 11 too.

Did you successfully run inference in TensorRT c++ verison for any tensorflow object detection models?? custom trained models.

No, I use the python-api of TensorRT. (But I created some working custom-trained TensorRT engines for object detection.)

Python api works well with custom trained ssd inception models.

C++ is not producing any detections. also not giving any error messgaes for me.

I did tensorflow - UFF - tensorRT (python works, c++ does not work)

I do not know what could be the problem. because there are no error messages.

@AastaLLL @joel.oswald @kayccc,
I got this error after the above solution mentioned by @joel.oswald
it seems NonZero plugin is missing.
But how io write a custom plugin for it…
Any work around for this???

Any idea

[08/10/2020-14:16:52] [V] [TRT] ModelImporter.cpp:103: Parsing node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater [Greater]
[08/10/2020-14:16:52] [V] [TRT] ModelImporter.cpp:119: Searching for input: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Reshape_9:0
[08/10/2020-14:16:52] [V] [TRT] ModelImporter.cpp:119: Searching for input: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_42/Greater/y:0
[08/10/2020-14:16:52] [V] [TRT] ModelImporter.cpp:125: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater [Greater] inputs: [Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Reshape_9:0 -> (-1)], [Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_42/Greater/y:0 -> ()],
[08/10/2020-14:16:52] [V] [TRT] ImporterContext.hpp:141: Registering layer: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater for ONNX node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater
[08/10/2020-14:16:52] [V] [TRT] ImporterContext.hpp:116: Registering tensor: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater:0 for ONNX tensor: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater:0
[08/10/2020-14:16:52] [V] [TRT] ModelImporter.cpp:179: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater [Greater] outputs: [Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater:0 -> (-1)],
[08/10/2020-14:16:52] [V] [TRT] ModelImporter.cpp:103: Parsing node: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Where [NonZero]
[08/10/2020-14:16:52] [V] [TRT] ModelImporter.cpp:119: Searching for input: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater:0
[08/10/2020-14:16:52] [V] [TRT] ModelImporter.cpp:125: Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Where [NonZero] inputs: [Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan_9/Greater:0 -> (-1)],
[08/10/2020-14:16:52] [I] [TRT] ModelImporter.cpp:135: No importer registered for op: NonZero. Attempting to import as plugin.
[08/10/2020-14:16:52] [I] [TRT] builtin_op_importers.cpp:3659: Searching for plugin: NonZero, plugin_version: 1, plugin_namespace:
[08/10/2020-14:16:52] [E] [TRT] INVALID_ARGUMENT: getPluginCreator could not find plugin NonZero version 1
ERROR: builtin_op_importers.cpp:3661 In function importFallbackPluginImporter:
[8] Assertion failed: creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[08/10/2020-14:16:52] [E] Failed to parse onnx file
[08/10/2020-14:16:52] [E] Parsing model failed
[08/10/2020-14:16:52] [E] Engine creation failed
[08/10/2020-14:16:52] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # ./trtexec --onnx=ssd_frozen.onnx --verbose

Hi All, @AastaLLL,

I am facing the same problem. I have trained the SSD_mobilenet_v2_coco model on my own dataset and converted it to onnx using tf2onnx --opset 11.

The onnx file creation works fine. But when I try to run this on DeepStream 5.0, I get the UINT8 error when it tries to make the engine file.

Even the onnx file converted directly from the pretrained model give the UINT8 error.

Any help would be welcome!

Hi, anujfulari

We try your model but meet some issue in the output dimension.

May I know the class number of your model?
We try the default 91 value and cannot match the dimension.

Thanks.

Hi @AastaLLL,
Number of classes in the model is 3 excluding background class.

Hi, anujfulari

We can run your model with the configure file shared in this topic but update the class number into 4.

diff --git a/config.py b/config.py
index 499a605..444af99 100644
--- a/config.py
+++ b/config.py
@@ -36,7 +36,7 @@ NMS = gs.create_plugin_node(name="NMS", op="NMS_TRT",
     nmsThreshold=0.6,
     topK=100,
     keepTopK=100,
-    numClasses=3,
+    numClasses=4,
     inputOrder=[0, 2, 1],
     confSigmoid=1,
     isNormalized=1)
$ sudo python3 /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py frozen_inference_graph.pb -o sample_ssd_relu6.uff -O NMS -p config.py
$ /usr/src/tensorrt/bin/trtexec --uff=./sample_ssd_relu6.uff --uffInput=Input,3,300,300 --output=NMS

Please give it a try and let us know the following.
Thanks.

Thank you @AastaLLL, for your help.
It successfully worked using uff model. I have tried it. But it gives error when we try to use ONNX method.
We were unable to convert ONNX model to TensorRT engine.

Thank you again.

Hi,

If the files are generated from the same TensorFlow model, you can use the .uff directly.
The problem in onnx is non-supported data format, which is not supported by the current TensorRT yet.

Thanks.

Ok.
Thank you for your support.

Hi everyone. I am currently trying the solution by @joel.oswald. I generated a fixed version of the frozen inference graph, but now I need to know the name of the output layer for my network. How do I get that? I am working with the TF Object Detection API.

Alternatively, we could try and fix the contents of the saved_model directory, which make it unnecessary to pass the --inputs and --outputs parameters to the ONNX conversion script. Does anyone know how to do that?

I was able to print the output of a model from the TF Object Detection API using graph surgeon:

import graphsurgeon as gs
graph = gs.DynamicGraph('/PATH/TO/FROZEN_GRAPH.pb')

print(graph.graph_outputs)
[name: "detection_boxes"
 op: "Identity"
 input: "Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayStack/TensorArrayGatherV3"
 attr {
   key: "T"
   value {
     type: DT_FLOAT
   }
 },
 name: "detection_scores"
 op: "Identity"
 input: "Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayStack_1/TensorArrayGatherV3"
 attr {
   key: "T"
   value {
     type: DT_FLOAT
   }
 },
 name: "detection_multiclass_scores"
 op: "Identity"
 input: "Postprocessor/BatchMultiClassNonMaxSuppression/map/TensorArrayStack_5/TensorArrayGatherV3"
 attr {
   key: "T"
   value {
     type: DT_FLOAT
   }
 },
 name: "detection_classes"
 op: "Identity"
 input: "add"
 attr {
   key: "T"
   value {
     type: DT_FLOAT
   }
 },
 name: "num_detections"
 op: "Identity"
 input: "Postprocessor/Cast_4"
 attr {
   key: "T"
   value {
     type: DT_FLOAT
   }
 },
 name: "raw_detection_boxes"
 op: "Identity"
 input: "Postprocessor/Squeeze"
 attr {
   key: "T"
   value {
     type: DT_FLOAT
   }
 },
 name: "raw_detection_scores"
 op: "Identity"
 input: "Postprocessor/convert_scores"
 attr {
   key: "T"
   value {
     type: DT_FLOAT
   }
 }]

However, I don’t know which of the outputs I should pass to the --outputs option in python -m tf2onnx.convert. Any advice?

Hi,

I will use ‘num_detections’, ‘detection_classes’, ‘detection_boxes’ and ‘detection_scores’.
Thanks.

And so did you succeed to convert onnx to tensorrt for ssd mobilenet model?
I have the same error when converting onnx to tensorrt.
I successfully generated onnx file, but failed to convert onnx to tensorrt.
Please give me help.

Hi akulov.eugen,

Please help to open a new topic for your issue. Thanks

Do you want me to open new issue with my problem?