Need help creating custom NMSPlugin.cpp

rms45 · August 12, 2020, 2:13pm

Description

I am currently trying to optimize SSDMobilenetV2 for inference speed on the Jetson TX2 by pruning anchor boxes that are not used by the classes in my dataset. The initial number of anchor boxes in the default implementation of SSDMobilenetV2 from the Tensorflow object detection zoo is 1917. And after pruning the number of boxes comes to 1885. Not a significant decrease but still I plan to do more optimizations later on and hence this info might be useful.

I am successfully able to convert the default SSDMobilenetV2 model to a TensorRT binary with the plugins provided by TensorRT, but I cannot convert the custom model. The compilation stops by throwing an aassertion error at line 246 of NMSPlugin.cpp. This is the line which checks the following:

ASSERT(numPriors * numLocClasses * nbBoxCoordinates == inputDims[param.inputOrder[0]].d[0]);

Printing the values of these variables for different input orders (given in ) I get,

pruned:[021]   numPriors:0    numLocClasses:1 C1:2    C2:5655 C3:1
pruned:[012]   numPriors:0    numLocClasses:1 C1:2    C2:7540 C3:1
pruned:[102]   numPriors:0    numLocClasses:1 C1:7540 C2:2    C3:1
pruned:[120]   numPriors:1917 numLocClasses:1 C1:7540 C2:5655 C3:7668
pruned:[201]   numPriors:0    numLocClasses:1 C1:5655 C2:2    C3:1
pruned:[210]   numPriors:1917 numLocClasses:1 C1:5655 C2:7540 C3:7668

The right value of numPriors should be 1885 (which is 7540/4). But I get 1917(which is 7668/4) which is the older number of boxes. Why do I get the older value of anchor boxes here?
I do not understand from where inputDims gets its values. Is this configurable?

Environment

TensorRT Version: 7.1.0
GPU Type: PASCAL
Nvidia Driver Version:
CUDA Version: 10.2
CUDNN Version: 8.0
Operating System + Version: L4T from JetPack 4.4 DP
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): 1.15.0
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
https://drive.google.com/file/d/14AJL87iLsFkrt4qLIfS-m8lkmUVvpD_g/view?usp=sharing

Steps To Reproduce

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

Log when using trtexec:

...
[08/09/2020-15:57:13] [V] [TRT] Plugin creator registration succeeded - ::Split
[08/09/2020-15:57:13] [V] [TRT] Plugin creator registration succeeded - ::SpecialSlice_TRT
[08/09/2020-15:57:13] [V] [TRT] Plugin creator registration succeeded - ::InstanceNormalization_TRT
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Parsing MultipleGridAnchorGenerator[Op: _GridAnchor_TRT].
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Parsing concat_priorbox[Op: Concat]. Inputs: MultipleGridAnchorGenerator
[08/09/2020-15:57:14] [V] [TRT] UFFParser: concat_priorbox -> [2,7668,1]
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Applying order forwarding to: concat_priorbox
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Parsing Input[Op: Input].
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Input -> [1,3,300,300]
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Applying order forwarding to: Input
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Parsing FeatureExtractor/MobilenetV2/Conv/weights[Op: Const].
[08/09/2020-15:57:14] [V] [TRT] UFFParser: FeatureExtractor/MobilenetV2/Conv/weights -> [3,3,3,32]
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Applying order forwarding to: FeatureExtractor/MobilenetV2/Conv/weights
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Parsing FeatureExtractor/MobilenetV2/Conv/Conv2D[Op: Conv]. Inputs: Input, FeatureExtractor/MobilenetV2/Conv/weights
[08/09/2020-15:57:14] [V] [TRT] UFFParser: Inserting transposes for FeatureExtractor/MobilenetV2/Conv/Conv2D
[08/09/2020-15:57:14] [E] [TRT] UffParser: Parser error: FeatureExtractor/MobilenetV2/Conv/Conv2D: Order size is not matching the number dimensions of TensorRT
[08/09/2020-15:57:14] [E] Failed to parse uff file
[08/09/2020-15:57:14] [E] Parsing model failed
[08/09/2020-15:57:14] [E] Engine creation failed
[08/09/2020-15:57:14] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --uff=tmp_v2_coco.uff --uffInput=image_tensor:0,3,300,300 --output=NMS:0 --fp16 --verbose --saveEngine=trt.bin

Log using build_engine.py:

python build_engine.py
…
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/expanded_conv_13/expand/Conv2D + FeatureExtractor/MobilenetV2/expanded_conv_13/expand/Relu6 (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x64_relu_interior_nn_v1
[TensorRT] VERBOSE: BoxPredictor_0/ClassPredictor/Conv2D || BoxPredictor_0/BoxEncodingPredictor/Conv2D (hcudnn_winograd) Set Tactic Name: maxwell_fp16x2_hcudnn_winograd_fp16x2_128x128_ldg1_ldg4_relu_tile148m_nt_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/expanded_conv_13/project/Conv2D (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_interior_nn_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/expanded_conv_14/expand/Conv2D + FeatureExtractor/MobilenetV2/expanded_conv_14/expand/Relu6 (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_interior_nn_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/expanded_conv_14/project/Conv2D + FeatureExtractor/MobilenetV2/expanded_conv_14/add (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_interior_nn_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/expanded_conv_15/expand/Conv2D + FeatureExtractor/MobilenetV2/expanded_conv_15/expand/Relu6 (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_interior_nn_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/expanded_conv_15/project/Conv2D + FeatureExtractor/MobilenetV2/expanded_conv_15/add (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_interior_nn_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/expanded_conv_16/expand/Conv2D + FeatureExtractor/MobilenetV2/expanded_conv_16/expand/Relu6 (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_interior_nn_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/expanded_conv_16/project/Conv2D (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x64_relu_interior_nn_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/Conv_1/Conv2D + FeatureExtractor/MobilenetV2/Conv_1/Relu6 (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x128_relu_interior_nn_v1
[TensorRT] VERBOSE: BoxPredictor_1/ClassPredictor/Conv2D || BoxPredictor_1/BoxEncodingPredictor/Conv2D (hcudnn_winograd) Set Tactic Name: maxwell_fp16x2_hcudnn_winograd_fp16x2_128x128_ldg1_ldg4_relu_tile148m_nt_v1
[TensorRT] VERBOSE: FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/Conv2D + FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/Relu6 (hcudnn) Set Tactic Name: maxwell_fp16x2_hcudnn_fp16x2_128x64_relu_interior_nn_v1
#assertionnmsPlugin.cpp,246
Aborted (core dumped)

AakankshaS · August 12, 2020, 8:51pm

Hi @rms45

This error states that there might be a mismatch between the training data and the deployment dimensions
However you may find help from this relevant post.

Thanks!

rms45 · August 14, 2020, 8:30pm

Hi @AakankshaS,
Thank you for the response. But I have already seen this post. I do not think this is relevant in my case. Neither am I using a wrong output name such as MarkOutput or using a batch dimension. Also as i said earlier, I was able to build the unmodified version of SSD to a TensorRT binary without any problems. However when I change the number of anchor boxes, I get this error. I have not made any other changes to the model. So I think the problem lies elsewhere.

AakankshaS · August 18, 2020, 7:33pm

Hi @rms45,
Can you try the onnx conversion instead of uff?

Thanks!

rms45 · August 24, 2020, 11:54am

Using ONNX results in the following error:
[TensorRT] VERBOSE: Plugin creator already registered - ::GridAnchor_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::NMS_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::Reorg_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::Region_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::Clip_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::LReLU_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::PriorBox_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::Normalize_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::RPROI_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::BatchedNMS_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::FlattenConcat_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::CropAndResize
[TensorRT] VERBOSE: Plugin creator already registered - ::DetectionLayer_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::Proposal
[TensorRT] VERBOSE: Plugin creator already registered - ::ProposalLayer_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::PyramidROIAlign_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::ResizeNearest_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::Split
[TensorRT] VERBOSE: Plugin creator already registered - ::SpecialSlice_TRT
[TensorRT] VERBOSE: Plugin creator already registered - ::InstanceNormalization_TRT
WARNING:tensorflow:From onnx_from_tf.py:24: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

[19, 10, 5, 3, 2, 1]
WARNING:tensorflow:From /home/vanderlande/.venvs/optimizationofdlmodels/lib/python3.6/site-packages/graphsurgeon/node_manipulation.py:106: The name tf.NodeDef is deprecated. Please use tf.compat.v1.NodeDef instead.

WARNING: To create TensorRT plugin nodes, please use the create_plugin_node function instead.
Traceback (most recent call last):
  File "onnx_from_tf.py", line 36, in <module>
    g = tf.import_graph_def(g2.as_graph_def(), name='')
  File "/home/vanderlande/.venvs/optimizationofdlmodels/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/vanderlande/.venvs/optimizationofdlmodels/lib/python3.6/site-packages/tensorflow_core/python/framework/importer.py", line 405, in import_graph_def
    producer_op_list=producer_op_list)
  File "/home/vanderlande/.venvs/optimizationofdlmodels/lib/python3.6/site-packages/tensorflow_core/python/framework/importer.py", line 501, in _import_graph_def_internal
    graph._c_graph, serialized, options)  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.NotFoundError: Op type not registered 'GridAnchor_TRT' in binary running on JetsonTx2. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.

This is related to issue tf-onnx error using tf.contrib · Issue #768 · onnx/tensorflow-onnx · GitHub. It seems if tf.contrib is used, the onnx conversion does not work.

AakankshaS · August 24, 2020, 1:06pm

Hi @rms45,
Can you please share your onnx model?

Thanks!

rms45 · August 24, 2020, 1:30pm

This error occurs before I can convert the model to ONNX. It seems UFF is the only option for now.

AakankshaS · August 25, 2020, 5:28pm

Hi @rms45,

The reason i suggested you to try ONNX is because UFF parser is deprecated from TRT 7 onwards, hence we plan to remove the support in the subsequent major release.
There is a suggestion given in the post you have shared, which might be the possible solution
https://github.com/onnx/tensorflow-onnx/issues/768#issuecomment-585003968

Thanks!

rms45 · August 31, 2020, 2:39pm

Hi @AakankshaS,
Yes but ONNX does not support all the ops for SSD currently. And UFF does support it because I was able to use it to convert a generic model. I just wanted to know from where the values of numPriors, numLocClasses are passed on to the plugin. Is there a way to visualize the tensorrt graph with the plugins so I can debug this?

gustcool2 · March 11, 2021, 1:01am

Hi @rms45.
I have the same issue.
I trained own datasets(ssd mobilenet v2)
and ran it using TRT_object_detection(GitHub - AastaNV/TRT_object_detection: Python sample for referencing object detection model with TensorRT)
but, it didn’t work well.
#assertionnmsPlugin.cpp,246
please help.

sahil24wadhwa · May 17, 2022, 5:52am

I am also getting similar error. #assertion/opt/tensorrt/TensorRT/plugin/nmsPlugin/nmsPlugin.cpp,82

Where can I find the nmsPlugin.cpp file?

Topic		Replies	Views
Problem converting ONNX model to TensorRT Engine for SSD Mobilenet V2 Jetson Nano tensorrt , nvbugs , ssd , onnx	38	8769	October 18, 2021
UFFParser: Parser error: BoxPredictor_0/Reshape: Reshape: -1 dimension specified more than 1 time Jetson Nano tensorrt	9	887	October 18, 2021
TensorRT conversion from tensorflow with custom op TensorRT tensorrt , tensorflow	5	1340	August 12, 2023
Onnx to TensorRT, with NVIDIA plugins (gridAnchor) TensorRT	11	2110	November 16, 2021
Conversion to tensorRT error . [graphShapeAnalyzer.cpp::throwIfError::1306] Error Code 9 TensorRT jetson-inference	10	4359	May 13, 2022
Process killed during tensorrt conversion on Jetson orin NX (8 GB) Jetson Orin NX tensorrt	15	725	April 30, 2024
I am trying to convert the ONNX SSD mobilnet v3 model into TensorRT Engine. I am getting the below error Jetson TX2 tensorrt , tensorflow	24	3702	February 17, 2022
Tensorrt support for SSD_inception trained on custom dataset TensorRT	15	2619	October 12, 2021
Cannot convert SSD ONNX model to TensorRT TensorRT tensorrt	15	2359	November 23, 2022
Running a pytorch network converted to ONNX with TensorRT on the TX2 Jetson TX2	24	8879	October 18, 2021

Need help creating custom NMSPlugin.cpp

Description

Environment

Relevant Files

Steps To Reproduce

Related topics