Trtexec create engine failed from onnx when adding dynamic shapes

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I’m using trtexec to create engine for efficientnet-b0.

First I converted my pytorch model to onnx format with static shapes and then converted to trt engine, everything is OK at this time.

Then I tried to add dynamic shapes, here is the conversion code.

x = torch.randn(1, 3, 224, 224).cuda()
dynamic_axes= {'input':{0:'batch_size' , 2:'width', 3:'height'}, 'output':{0:'batch_size' , 2:'width', 3:'height'}}
torch.onnx.export(model, x, "test.onnx", opset_version=10, do_constant_folding=True, input_names=['input'], output_names=['output'], dynamic_axes=dynamic_axes)

But I failed to convert this onnx to trt engine with the following two commands.

/data/TensorRT-6.0.1.5/bin/trtexec --onnx=test.onnx --minShapes=input:1x3x224x224 --optShapes=input:10x3x224x224 --maxShapes=input:30x3x224x224 --shapes=input:5x3x224x224 --saveEngine=test.trt

/data/TensorRT-6.0.1.5/bin/trtexec --onnx=test.onnx --saveEngine=test.trt

Traceback of errors:

[05/17/2021-12:16:24] [I] === Model Options ===
[05/17/2021-12:16:24] [I] Format: ONNX
[05/17/2021-12:16:24] [I] Model: test.onnx
[05/17/2021-12:16:24] [I] Output:
[05/17/2021-12:16:24] [I] === Build Options ===
[05/17/2021-12:16:24] [I] Max batch: explicit
[05/17/2021-12:16:24] [I] Workspace: 16 MB
[05/17/2021-12:16:24] [I] minTiming: 1
[05/17/2021-12:16:24] [I] avgTiming: 8
[05/17/2021-12:16:24] [I] Precision: FP32
[05/17/2021-12:16:24] [I] Calibration: 
[05/17/2021-12:16:24] [I] Safe mode: Disabled
[05/17/2021-12:16:24] [I] Save engine: test.trt
[05/17/2021-12:16:24] [I] Load engine: 
[05/17/2021-12:16:24] [I] Inputs format: fp32:CHW
[05/17/2021-12:16:24] [I] Outputs format: fp32:CHW
[05/17/2021-12:16:24] [I] Input build shape: input=1x3x224x224+10x3x224x224+30x3x224x224
[05/17/2021-12:16:24] [I] === System Options ===
[05/17/2021-12:16:24] [I] Device: 0
[05/17/2021-12:16:24] [I] DLACore: 
[05/17/2021-12:16:24] [I] Plugins:
[05/17/2021-12:16:24] [I] === Inference Options ===
[05/17/2021-12:16:24] [I] Batch: Explicit
[05/17/2021-12:16:24] [I] Iterations: 10 (200 ms warm up)
[05/17/2021-12:16:24] [I] Duration: 10s
[05/17/2021-12:16:24] [I] Sleep time: 0ms
[05/17/2021-12:16:24] [I] Streams: 1
[05/17/2021-12:16:24] [I] Spin-wait: Disabled
[05/17/2021-12:16:24] [I] Multithreading: Enabled
[05/17/2021-12:16:24] [I] CUDA Graph: Disabled
[05/17/2021-12:16:24] [I] Skip inference: Disabled
[05/17/2021-12:16:24] [I] === Reporting Options ===
[05/17/2021-12:16:24] [I] Verbose: Disabled
[05/17/2021-12:16:24] [I] Averages: 10 inferences
[05/17/2021-12:16:24] [I] Percentile: 99
[05/17/2021-12:16:24] [I] Dump output: Disabled
[05/17/2021-12:16:24] [I] Profile: Disabled
[05/17/2021-12:16:24] [I] Export timing to JSON file: 
[05/17/2021-12:16:24] [I] Export profile to JSON file: 
[05/17/2021-12:16:24] [I] 
----------------------------------------------------------------
Input filename:   test.onnx
ONNX IR version:  0.0.4
Opset version:    10
Producer name:    pytorch
Producer version: 1.2
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
WARNING: ONNX model has a newer ir_version (0.0.4) than this parser was built against (0.0.3).
[05/17/2021-12:16:25] [E] [TRT] Parameter check failed at: ../builder/Network.cpp::addInput::671, condition: isValidDims(dims, hasImplicitBatchDimension())
ERROR: ModelImporter.cpp:80 In function importInput:
[8] Assertion failed: *tensor = importer_ctx->network()->addInput( input.name().c_str(), trt_dtype, trt_dims)
[05/17/2021-12:16:25] [E] Failed to parse onnx file
[05/17/2021-12:16:25] [E] Parsing model failed
[05/17/2021-12:16:25] [E] Engine could not be created

Environment

TensorRT Version: 6.0.1.5
GPU Type: T4
Nvidia Driver Version:
CUDA Version: 10.1
CUDNN Version: 7.6.5
Operating System + Version: CentOS 7
Python Version (if applicable): 3.7.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.2.0
Baremetal or Container (if container which image + tag):

Can anybody help? Thanks

2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

Hi,

I’ve tried to test my onnx model with your code and no errors reported.
Here is my onnx model.
test.onnx (15.5 MB)

5

Hi @751180903,

Looks like you’re using old version of TensorRT, we recommend you to try on latest TensorRT version.
When we tried building engine using shared ONNX file on latest version, unable to reproduce the error. Engine generated successfully.

For your reference, to upgrade TensorRT

Thank you.

6

Hi,

I’ve tested carefully the model on version 6.0.1.5 and I found that 6.0.1.5 only supports dynamic batches. Dynamic shapes which means height and width are not supported.

7

@751180903,

As we suggested, we recommend you to try on latest TensorRT version.
Also please refer optimization profiles regarding dynamic shapes.

Thank you.