Build engine from onnx model, can not set dynamic shape right

[AaronCoC]

Description

I try to export my onnx(set dynmiac axes already) model to trt engine with dynamic shapes. I try to configured optimized profile to set the dynamic shapes, but failed.

Environment

TensorRT Version: 8…6.1
GPU Type: RTX3090
Nvidia Driver Version: 11.3
CUDA Version: 11.7
CUDNN Version:
Operating System + Version: ubuntu 20.04
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.12
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
code:
test_export_trt.py (4.5 KB)

model:
decoder.onnx (2.4 MB)

Steps To Reproduce

Change the onnx_model_path and run, check engine.get_profile_shapes and profile.get_shape if are same

Please include:

• Exact steps/commands to build your repro
• Exact steps/commands to run your repro
• Full traceback of errors encountered

[AakankshaS]

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

Quick Start Guide :: NVIDIA Deep Learning TensorRT Documentation

1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

TensorRT/samples/trtexec at master · NVIDIA/TensorRT

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

[AaronCoC]

The model and code I’ve already sent in Relevant Files block.
And I tried trtexec to convert the onnx, if I not set the min/opt/max dynamic shapes with command

${trtexec} --onnx=output/models/${model_name}.onnx \
    --saveEngine=output/models/${model_name}.engine \
    --dumpOutput \
    --dumpProfile \
    --dumpLayerInfo --useSpinWait \
    --separateProfileRun >output/models/log/trtexec.log

I will got:

If I set the dynamic shapes with command:

min_batch=1
max_batch=8
opt_batch=4

${trtexec} --onnx=output/models/${model_name}.onnx \
    --saveEngine=output/models/${model_name}.engine \
    --minShapes=element_states_batch:${min_batch}x136x128,element_hidden_states:${min_batch}x136x128,element_states_mask:${min_batch}x136,lane_states_batch:${min_batch}x128x128,lane_mask:${min_batch}x128,sparse_goals_2D:${min_batch}x1536x2,sparse_goals_2D_mask:${min_batch}x1536 \
    --optShapes=element_states_batch:${opt_batch}x136x128,element_hidden_states:${opt_batch}x136x128,element_states_mask:${opt_batch}x136,lane_states_batch:${opt_batch}x128x128,lane_mask:${opt_batch}x128,sparse_goals_2D:${opt_batch}x1536x2,sparse_goals_2D_mask:${opt_batch}x1536 \
    --maxShapes=element_states_batch:${max_batch}x136x128,element_hidden_states:${max_batch}x136x128,element_states_mask:${max_batch}x136,lane_states_batch:${max_batch}x128x128,lane_mask:${max_batch}x128,sparse_goals_2D:${max_batch}x1536x2,sparse_goals_2D_mask:${max_batch}x1536 \
    --dumpOutput \
    --dumpProfile \
    --dumpLayerInfo --useSpinWait \
    --separateProfileRun >output/models/log/trtexec.log

And I got this:

Whole trtexec log:
trtexec.log (10.0 KB)

[spolisetty]

Hi,

Sorry for the delayed response, Are you still facing this issue?

Thank you.

[AaronCoC]

That’s okay,
Already found operator problem and replace it, thanks for reply.