My Yolov4 tf2checkpoint->save_model->onnx->trt

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:125: StatefulPartitionedCall/functional_3/tf_op_layer_Tile_6/Tile_6 [Tile] inputs: [StatefulPartitionedCall/functional_3/tf_op_layer_ExpandDims/ExpandDims:0 → (-1, -1, -1, -1)], [const_fold_opt__444 → (4)],

[11/01/2021-13:17:13] [V] [TRT] ImporterContext.hpp:141: Registering layer: StatefulPartitionedCall/functional_3/tf_op_layer_Tile_6/Tile_6 for ONNX node: StatefulPartitionedCall/functional_3/tf_op_layer_Tile_6/Tile_6

[11/01/2021-13:17:13] [V] [TRT] ImporterContext.hpp:116: Registering tensor: tf_op_layer_Tile_6 for ONNX tensor: tf_op_layer_Tile_6

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:179: StatefulPartitionedCall/functional_3/tf_op_layer_Tile_6/Tile_6 [Tile] outputs: [tf_op_layer_Tile_6 → (-1, -1, -1, -1)],

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:103: Parsing node: [BatchedNMS_TRT]

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:119: Searching for input: tf_op_layer_Tile_6

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:119: Searching for input: tf_op_layer_strided_slice_73

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:125: [BatchedNMS_TRT] inputs: [tf_op_layer_Tile_6 → (-1, -1, -1, -1)], [tf_op_layer_strided_slice_73 → (-1, -1, -1)],

[11/01/2021-13:17:13] [I] [TRT] ModelImporter.cpp:135: No importer registered for op: BatchedNMS_TRT. Attempting to import as plugin.

[11/01/2021-13:17:13] [I] [TRT] builtin_op_importers.cpp:3659: Searching for plugin: BatchedNMS_TRT, plugin_version: 1, plugin_namespace:

[11/01/2021-13:17:13] [I] [TRT] builtin_op_importers.cpp:3676: Successfully created plugin: BatchedNMS_TRT

[11/01/2021-13:17:13] [V] [TRT] ImporterContext.hpp:141: Registering layer: (Unnamed Layer* 1307) [PluginV2Ext] for ONNX node:

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [V] [TRT] ImporterContext.hpp:116: Registering tensor: num_detections_1 for ONNX tensor: num_detections

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [V] [TRT] ImporterContext.hpp:116: Registering tensor: nmsed_boxes_1 for ONNX tensor: nmsed_boxes

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [V] [TRT] ImporterContext.hpp:116: Registering tensor: nmsed_scores_1 for ONNX tensor: nmsed_scores

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [V] [TRT] ImporterContext.hpp:116: Registering tensor: nmsed_classes_1 for ONNX tensor: nmsed_classes

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:179: [BatchedNMS_TRT] outputs: [num_detections → ()], [nmsed_boxes → ()], [nmsed_scores → ()], [nmsed_classes → ()],

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:507: Marking num_detections_1 as output: num_detections

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:507: Marking nmsed_boxes_1 as output: nmsed_boxes

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:507: Marking nmsed_scores_1 as output: nmsed_scores

[11/01/2021-13:17:13] [V] [TRT] ModelImporter.cpp:507: Marking nmsed_classes_1 as output: nmsed_classes

----- Parsing of ONNX model www3_fix.onnx.nms_test.onnx is Done ----

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [E] [TRT] (Unnamed Layer* 1307) [PluginV2Ext]: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.

[11/01/2021-13:17:13] [E] [TRT] Layer (Unnamed Layer* 1307) [PluginV2Ext] failed validation

[11/01/2021-13:17:13] [E] [TRT] Network validation failed.

[11/01/2021-13:17:13] [E] Engine creation failed

[11/01/2021-13:17:13] [E] Engine set up failed

&&&& FAILED TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --onnx=www3_fix.onnx.nms_test.onnx --fp16 --saveEngine=add_BatchedNMS_TRT.engine --shapes=images:0:2x416x416x3 --verbose --explicitBatch

I try to convert the save_model to trt, after convert to onnx, i add BatchedNMS_TRT ;

#!/usr/bin/env python3
import onnx_graphsurgeon as gs
import argparse
import onnx
import numpy as np

def create_and_add_plugin_node(graph, topK, keepTopK):
    
    batch_size = graph.inputs[0].shape[0]
    #batch_size = 2
    input_h = graph.inputs[0].shape[1]
    input_w = graph.inputs[0].shape[2]
    print("*******(batch_size)********:", graph.inputs[0].shape[0])
    print("*******(input_h)********:", graph.inputs[0].shape[1])
    print("*******(input_w)********:", graph.inputs[0].shape[2])
    tensors = graph.tensors()
    boxes_tensor = tensors["tf_op_layer_Tile_6"]
    confs_tensor = tensors["tf_op_layer_strided_slice_73"]
    #boxes_tensor = tensors["boxes"]
    #confs_tensor = tensors["confs"]
    
    num_detections = gs.Variable(name="num_detections").to_variable(dtype=np.int32, shape=[batch_size, 1])
    nmsed_boxes = gs.Variable(name="nmsed_boxes").to_variable(dtype=np.float32, shape=[batch_size, keepTopK, 4])
    nmsed_scores = gs.Variable(name="nmsed_scores").to_variable(dtype=np.float32, shape=[batch_size, keepTopK])
    nmsed_classes = gs.Variable(name="nmsed_classes").to_variable(dtype=np.float32, shape=[batch_size, keepTopK])

    new_outputs = [num_detections, nmsed_boxes, nmsed_scores, nmsed_classes]

    mns_node = gs.Node(
        op="BatchedNMS_TRT",
        #op="BatchedNMSDynamic_TRT",
        attrs=create_attrs(input_h, input_w, topK, keepTopK),
        inputs=[boxes_tensor, confs_tensor],
        outputs=new_outputs)

    graph.nodes.append(mns_node)
    graph.outputs = new_outputs

    return graph.cleanup().toposort()




def create_attrs(input_h, input_w, topK, keepTopK):

    num_anchors = 3

    h1 = input_h // 8
    h2 = input_h // 16
    h3 = input_h // 32

    w1 = input_w // 8
    w2 = input_w // 16
    w3 = input_w // 32

    num_boxes = num_anchors * (h1 * w1 + h2 * w2 + h3 * w3)

    attrs = {}

    attrs["shareLocation"] = 1
    attrs["backgroundLabelId"] = -1
    attrs["numClasses"] = 2
    attrs["topK"] = topK
    attrs["keepTopK"] = keepTopK
    attrs["scoreThreshold"] = 0.5
    attrs["iouThreshold"] = 0.6
    attrs["isNormalized"] = 1
    attrs["clipBoxes"] = 1

    # 001 is the default plugin version the parser will search for, and therefore can be omitted,
    # but we include it here for illustrative purposes.
    attrs["plugin_version"] = "1"

    return attrs


def main():
    parser = argparse.ArgumentParser(description="Add batchedNMSPlugin")
    parser.add_argument("-f", "--model", help="Path to the ONNX model generated by export_model.py", default="yolov4_1_3_416_416.onnx")
    parser.add_argument("-t", "--topK", help="number of bounding boxes for nms", default=2000)
    parser.add_argument("-k", "--keepTopK", help="bounding boxes to be kept per image", default=1000)

    args, _ = parser.parse_known_args()

    graph = gs.import_onnx(onnx.load(args.model))
    
    graph = create_and_add_plugin_node(graph, int(args.topK), int(args.keepTopK))
    
    onnx.save(gs.export_onnx(graph), args.model + ".nms_test.onnx")


if __name__ == '__main__':
    main()
  • but when I convert the onnx to trt failed;
    /usr/src/tensorrt/bin/trtexec --onnx=www3_fix.onnx.nms_test.onnx --fp16 --saveEngine=add_BatchedNMS_TRT.engine --shapes=images:0:2x416x416x3 --verbose --explicitBatch

  • my Jetson’s TensorRT Version:7.1.3
    I am sure that the BatchedNMS’s input shape is [batch_size, number_boxes, number_classes, number_box_parameters]
    image

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:
https://docs.nvidia.com/deeplearning/tensorrt/quick-start-guide/index.html#onnx-export

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!