Pytorch model convert to TensorRT engine failed

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
1

Description

My workflow:
Pytorch->ONNX->TensorRT engine

I have checked that all of my layers is supported by TensorRT but still got some problem.

The verbose:

[09/16/2020-15:06:17] [I] === Build Options ===
[09/16/2020-15:06:17] [I] Max batch: explicit
[09/16/2020-15:06:17] [I] Workspace: 16 MB
[09/16/2020-15:06:17] [I] minTiming: 1
[09/16/2020-15:06:17] [I] avgTiming: 8
[09/16/2020-15:06:17] [I] Precision: FP32
[09/16/2020-15:06:17] [I] Calibration:
[09/16/2020-15:06:17] [I] Safe mode: Disabled
[09/16/2020-15:06:17] [I] Save engine:
[09/16/2020-15:06:17] [I] Load engine:
[09/16/2020-15:06:17] [I] Builder Cache: Enabled
[09/16/2020-15:06:17] [I] NVTX verbosity: 0
[09/16/2020-15:06:17] [I] Inputs format: fp32:CHW
[09/16/2020-15:06:17] [I] Outputs format: fp32:CHW
[09/16/2020-15:06:17] [I] Input build shape: 32x3x24x94=
[09/16/2020-15:06:17] [I] Input build shape: 1x3x24x94=1x3x24x94
[09/16/2020-15:06:17] [I] Input build shape: 16x3x24x94=
[09/16/2020-15:06:17] [I] Input calibration shapes: model
[09/16/2020-15:06:17] [I] === System Options ===
[09/16/2020-15:06:17] [I] Device: 0
[09/16/2020-15:06:17] [I] DLACore:
[09/16/2020-15:06:17] [I] Plugins:
[09/16/2020-15:06:17] [I] === Inference Options ===
[09/16/2020-15:06:17] [I] Batch: Explicit
[09/16/2020-15:06:17] [I] Input inference shape: 16x3x24x94=16x3x24x94
[09/16/2020-15:06:17] [I] Input inference shape: 32x3x24x94=
[09/16/2020-15:06:17] [I] Input inference shape: 1x3x24x94=
[09/16/2020-15:06:17] [I] Iterations: 10
[09/16/2020-15:06:17] [I] Duration: 3s (+ 200ms warm up)
[09/16/2020-15:06:17] [I] Sleep time: 0ms
[09/16/2020-15:06:17] [I] Streams: 1
[09/16/2020-15:06:17] [I] ExposeDMA: Disabled
[09/16/2020-15:06:17] [I] Spin-wait: Disabled
[09/16/2020-15:06:17] [I] Multithreading: Disabled
[09/16/2020-15:06:17] [I] CUDA Graph: Disabled
[09/16/2020-15:06:17] [I] Skip inference: Disabled
[09/16/2020-15:06:17] [I] Inputs:
[09/16/2020-15:06:17] [I] === Reporting Options ===
[09/16/2020-15:06:17] [I] Verbose: Enabled
[09/16/2020-15:06:17] [I] Averages: 10 inferences
[09/16/2020-15:06:17] [I] Percentile: 99
[09/16/2020-15:06:17] [I] Dump output: Disabled
[09/16/2020-15:06:17] [I] Profile: Disabled
[09/16/2020-15:06:17] [I] Export timing to JSON file:
[09/16/2020-15:06:17] [I] Export output to JSON file:
[09/16/2020-15:06:17] [I] Export profile to JSON file:
[09/16/2020-15:06:17] [I]
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::Proposal version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::Split version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[09/16/2020-15:06:17] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1

[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::GridAnchor_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::NMS_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::Reorg_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::Region_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::Clip_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::LReLU_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::PriorBox_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::Normalize_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::RPROI_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::FlattenConcat_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::CropAndResize version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::Proposal version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::Split version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 1
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:202: Adding network input: input with dtype: float32, dimensions: (-1, 3, 24, 94)
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:116: Registering tensor: input for ONNX tensor: input
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.0.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.0.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.1.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.1.running_mean
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.1.running_var
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.1.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.11.block.0.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.11.block.0.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.11.block.2.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.11.block.2.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.11.block.4.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.11.block.4.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.11.block.6.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.11.block.6.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.12.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.12.running_mean
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.12.running_var
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.12.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.16.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.16.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.17.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.17.running_mean
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.17.running_var
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.17.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.20.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.20.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.21.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.21.running_mean
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.21.running_var
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.21.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.4.block.0.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.4.block.0.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.4.block.2.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.4.block.2.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.4.block.4.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.4.block.4.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.4.block.6.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.4.block.6.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.5.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.5.running_mean
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.5.running_var
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.5.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.8.block.0.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.8.block.0.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.8.block.2.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.8.block.2.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.8.block.4.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.8.block.4.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.8.block.6.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.8.block.6.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.9.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.9.running_mean
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.9.running_var
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: backbone.9.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: container.0.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:90: Importing initializer: container.0.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:103: Parsing node: Conv_0 [Conv]
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: input
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: backbone.0.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: backbone.0.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:125: Conv_0 [Conv] inputs: [input → (-1, 3, 24, 94)], [backbone.0.weight → (64, 3, 3, 3)], [backbone.0.bias → (64)],
[09/16/2020-15:06:18] [V] [TRT] builtin_op_importers.cpp:450: Convolution input dimensions: (-1, 3, 24, 94)
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:141: Registering layer: Conv_0 for ONNX node: Conv_0
[09/16/2020-15:06:18] [V] [TRT] builtin_op_importers.cpp:533: Using kernel: (3, 3), strides: (1, 1), prepadding: (0, 0), postpadding: (0, 0), dilations: (1, 1), numOutputs: 64
[09/16/2020-15:06:18] [V] [TRT] builtin_op_importers.cpp:534: Convolution output dimensions: (-1, 64, 22, 92)
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 63 for ONNX tensor: 63
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:179: Conv_0 [Conv] outputs: [63 → (-1, 64, 22, 92)],
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:103: Parsing node: BatchNormalization_1 [BatchNormalization]
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: 63
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: backbone.1.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: backbone.1.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: backbone.1.running_mean
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: backbone.1.running_var
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:125: BatchNormalization_1 [BatchNormalization] inputs: [63 → (-1, 64, 22, 92)], [backbone.1.weight → (64)], [backbone.1.bias → (64)], [backbone.1.running_mean → (64)], [backbone.1.running_var → (64)],
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:141: Registering layer: BatchNormalization_1 for ONNX node: BatchNormalization_1
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 64 for ONNX tensor: 64
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:179: BatchNormalization_1 [BatchNormalization] outputs: [64 → (-1, 64, 22, 92)],
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:103: Parsing node: Relu_2 [Relu]
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: 64
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:125: Relu_2 [Relu] inputs: [64 → (-1, 64, 22, 92)],
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:141: Registering layer: Relu_2 for ONNX node: Relu_2
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 65 for ONNX tensor: 65
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:179: Relu_2 [Relu] outputs: [65 → (-1, 64, 22, 92)],
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:103: Parsing node: MaxPool_3 [MaxPool]
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: 65
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:125: MaxPool_3 [MaxPool] inputs: [65 → (-1, 64, 22, 92)],
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:141: Registering layer: MaxPool_3 for ONNX node: MaxPool_3
[09/16/2020-15:06:18] [E] [TRT] MaxPool_3: at least 5 dimensions are required for input.
[09/16/2020-15:06:18] [E] [TRT] MaxPool_3: at least 5 dimensions are required for input.
[09/16/2020-15:06:18] [V] [TRT] ImporterContext.hpp:116: Registering tensor: 66 for ONNX tensor: 66
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:179: MaxPool_3 [MaxPool] outputs: [66 → ()],
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:103: Parsing node: Conv_4 [Conv]
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: 66
[09/16/2020-15:06:18] [E] [TRT] MaxPool_3: at least 5 dimensions are required for input.
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: backbone.4.block.0.weight
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:119: Searching for input: backbone.4.block.0.bias
[09/16/2020-15:06:18] [V] [TRT] ModelImporter.cpp:125: Conv_4 [Conv] inputs: [66 → ()], [backbone.4.block.0.weight → (32, 64, 1, 1)], [backbone.4.block.0.bias → (32)],
[09/16/2020-15:06:18] [E] [TRT] MaxPool_3: at least 5 dimensions are required for input.
[09/16/2020-15:06:18] [V] [TRT] builtin_op_importers.cpp:450: Convolution input dimensions: ()
ERROR: builtin_op_importers.cpp:469 In function importConv:
[8] Assertion failed: nbSpatialDims == kernelWeights.shape.nbDims - 2
[09/16/2020-15:06:18] [E] Failed to parse onnx file
[09/16/2020-15:06:18] [E] Parsing model failed
[09/16/2020-15:06:18] [E] Engine creation failed
[09/16/2020-15:06:18] [E] Engine set up failed

Environment

TensorRT Version : 7.1.3
GPU Type : Jetson TX2
Nvidia Driver Version :
CUDA Version : 10.2
CUDNN Version : 8
Operating System + Version : Ubuntu 18.04

Relevant Files

onnx model:
https://drive.google.com/file/d/1kyqqfmsEtoHUYM02H6sxUv1hMskWFmE9/view?usp=sharing

Steps To Reproduce

Using trtexec to convert the model to tensorRT engine file. By using the following command.

sudo ./trtexec --onnx=TRT_test.onnx --explicitBatch --minShapes=1x3x24x94 --optShapes=16x3x24x94 --maxShapes=32x3x24x94 --verbose

2

Hi @jackgao0323,
I could reproduce the issue with your model.
Our team is looking into this. Please stay tuned.

Thanks!

3

Hi @jackgao0323,
How was this model generated?
It doesn’t look like valid ONNX, and ONNX-RT can’t seem to run it either
Can you please check the same.

Thanks!

5

Hi @AakankshaS,
My model is referring to this github License_Plate_Detection_Pytorch/LPRNet/model at master · xuexingyu24/License_Plate_Detection_Pytorch · GitHub. It have pytorch model inside. You can take a look. I converted the onnx model from this pytorch model by python.

Thanks.

6

Hi @jackgao0323,
Can you please follow the below link for conversion of your pytorch model
https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html

In case if the issue persist, kindly raise it in here.
Thanks!

7

Hi @AakankshaS,
I’ve followed the steps by the link before.

torch.onnx.export(model,
x,
onnx_file_name,
export_params=True,
opset_version=11,
do_constant_folding=True,
input_names=input_names, output_names=output_names,
dynamic_axes=dynamic_axes)

I’m using the code like above. I’ll raise the issue in the github thanks.

BR,
Jack Gao