Tf2 checkpoint->save_model->onnx->trt (onnx->trt failed)

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
#1

I try to add NMS in trt; then I use onnx_graphsurgeon creat and add BatchedNMS_TRT to the end of onnx; finally convert onnx->trt;
the Jetson TensorRT version:7.1.3
I am sure the BatchedNMS_TRT’s input shape are [batch_size, number_boxes, number_classes, number_box_parameters] and the scores input shape are [batch_size, number_boxes, number_classes];
image
but when I convert the onnx(with BatchedNMS_TRT) to trt failed;

[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:179: StatefulPartitionedCall/functional_3/tf_op_layer_Tile_6/Tile_6 [Tile] outputs: [tf_op_layer_Tile_6 -> (-1, -1, -1, -1)], 
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:103: Parsing node:  [BatchedNMS_TRT]
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:119: Searching for input: tf_op_layer_Tile_6
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:119: Searching for input: tf_op_layer_strided_slice_73
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:125:  [BatchedNMS_TRT] inputs: [tf_op_layer_Tile_6 -> (-1, -1, -1, -1)], [tf_op_layer_strided_slice_73 -> (-1, -1, -1)], 
[11/01/2021-14:20:08] [I] [TRT] ModelImporter.cpp:135: No importer registered for op: BatchedNMS_TRT. Attempting to import as plugin.
[11/01/2021-14:20:08] [I] [TRT] builtin_op_importers.cpp:3659: Searching for plugin: BatchedNMS_TRT, plugin_version: 1, plugin_namespace: 
[11/01/2021-14:20:08] [I] [TRT] builtin_op_importers.cpp:3676: Successfully created plugin: BatchedNMS_TRT
[11/01/2021-14:20:08] [V] [TRT] ImporterContext.hpp:141: Registering layer: (Unnamed Layer* 1307) [PluginV2Ext] for ONNX node: 
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [V] [TRT] ImporterContext.hpp:116: Registering tensor: num_detections_1 for ONNX tensor: num_detections
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [V] [TRT] ImporterContext.hpp:116: Registering tensor: nmsed_boxes_1 for ONNX tensor: nmsed_boxes
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [V] [TRT] ImporterContext.hpp:116: Registering tensor: nmsed_scores_1 for ONNX tensor: nmsed_scores
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [V] [TRT] ImporterContext.hpp:116: Registering tensor: nmsed_classes_1 for ONNX tensor: nmsed_classes
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:179:  [BatchedNMS_TRT] outputs: [num_detections -> ()], [nmsed_boxes -> ()], [nmsed_scores -> ()], [nmsed_classes -> ()], 
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:507: Marking num_detections_1 as output: num_detections
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:507: Marking nmsed_boxes_1 as output: nmsed_boxes
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:507: Marking nmsed_scores_1 as output: nmsed_scores
[11/01/2021-14:20:08] [V] [TRT] ModelImporter.cpp:507: Marking nmsed_classes_1 as output: nmsed_classes
 ----- Parsing of ONNX model www3_fix.onnx.nms_test.onnx is Done ---- 
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[11/01/2021-14:20:08] [E] [TRT] Layer (Unnamed Layer* 1307) [PluginV2Ext] failed validation
[11/01/2021-14:20:08] [E] [TRT] Network validation failed.
[11/01/2021-14:20:08] [E] Engine creation failed
[11/01/2021-14:20:08] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --onnx=www3_fix.onnx.nms_test.onnx --fp16 --saveEngine=add_BatchedNMS_TRT.engine --shapes=images:0:2x416x416x3 --verbose --explicitBatch
#2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:
https://docs.nvidia.com/deeplearning/tensorrt/quick-start-guide/index.html#onnx-export

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!