Failure to convert onnx model to trt (parsing failure/engine setup failure)

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I am trying to convert the following model to onnx, then convert it to .trt:

I import the pickled file then I convert it to onnx using the command:
torch.onnx.export(bfm_decoder,dummy_input,"bfm_noneck_v3.onnx",input_names=['R','offset','alpha_shp','alpha_exp'],output_names=['output'],opset_version=11)

This operation is provided in detail in the file:

The I try to convert to trt using trtexec, but the trt conversion fails. I used the commands, their results are provided at the end:
./trtexec --onnx=bfm_noneck_v3.onnx --saveEngine=bfm_noneck_v3.trt
./trtexec --onnx=bfm_noneck_v3.onnx --saveEngine=bfm_noneck_v3.trt --verbose
./trtexec --onnx=bfm_noneck_v3.onnx --saveEngine=bfm_noneck_v3.trt --shapes=R:3x3,offset:3x1,alpha_shp:40x1,alpha_exp:10x1

Environment

TensorRT Version: 8.6.0.12
GPU Type: NVIDIA GeForce RTX 2060 with Max-Q Design
Nvidia Driver Version: nvidia-driver-530
CUDA Version: 12.1
CUDNN Version:
Operating System + Version: Ubuntu 22.04
Python Version (if applicable): 3.9.12
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

bfm_noneck_v3.onnx (22.4 MB)

trtexec results:

# with --verbose flag:
[03/29/2023-11:09:50] [V] [TRT] Squeeze_3 [Squeeze] outputs: [onnx::Add_7 -> (3, 38365)[FLOAT]],  
[03/29/2023-11:09:50] [V] [TRT] Parsing node: Constant_4 [Constant]
[03/29/2023-11:09:50] [V] [TRT] Constant_4 [Constant] inputs:  
[03/29/2023-11:09:50] [V] [TRT] Constant_4 [Constant] outputs: [onnx::Add_8 -> (3, 38365)[FLOAT]],  
[03/29/2023-11:09:50] [V] [TRT] Parsing node: Add_5 [Add]
[03/29/2023-11:09:50] [V] [TRT] Searching for input: onnx::Add_8
[03/29/2023-11:09:50] [V] [TRT] Searching for input: onnx::Add_7
[03/29/2023-11:09:50] [V] [TRT] Add_5 [Add] inputs: [onnx::Add_8 -> (3, 38365)[FLOAT]], [onnx::Add_7 -> (3, 38365)[FLOAT]],  
[03/29/2023-11:09:50] [V] [TRT] Registering layer: onnx::Add_8 for ONNX node: onnx::Add_8
[03/29/2023-11:09:50] [V] [TRT] Registering layer: Add_5 for ONNX node: Add_5
[03/29/2023-11:09:50] [V] [TRT] Registering tensor: onnx::MatMul_9 for ONNX tensor: onnx::MatMul_9
[03/29/2023-11:09:50] [V] [TRT] Add_5 [Add] outputs: [onnx::MatMul_9 -> (3, 38365)[FLOAT]],  
[03/29/2023-11:09:50] [V] [TRT] Parsing node: MatMul_6 [MatMul]
[03/29/2023-11:09:50] [V] [TRT] Searching for input: R
[03/29/2023-11:09:50] [V] [TRT] Searching for input: onnx::MatMul_9
[03/29/2023-11:09:50] [V] [TRT] MatMul_6 [MatMul] inputs: [R -> (3, 3)[FLOAT]], [onnx::MatMul_9 -> (3, 38365)[FLOAT]],  
[03/29/2023-11:09:50] [V] [TRT] Registering layer: MatMul_6 for ONNX node: MatMul_6
[03/29/2023-11:09:50] [V] [TRT] Registering tensor: onnx::Add_10 for ONNX tensor: onnx::Add_10
[03/29/2023-11:09:50] [V] [TRT] MatMul_6 [MatMul] outputs: [onnx::Add_10 -> (3, 38365)[FLOAT]],  
[03/29/2023-11:09:50] [V] [TRT] Parsing node: Add_7 [Add]
[03/29/2023-11:09:50] [V] [TRT] Searching for input: onnx::Add_10
[03/29/2023-11:09:50] [V] [TRT] Searching for input: offset
[03/29/2023-11:09:50] [V] [TRT] Add_7 [Add] inputs: [onnx::Add_10 -> (3, 38365)[FLOAT]], [offset -> (3, 1)[FLOAT]],  
[03/29/2023-11:09:50] [V] [TRT] Registering layer: Add_7 for ONNX node: Add_7
[03/29/2023-11:09:50] [V] [TRT] Registering tensor: output_0 for ONNX tensor: output
[03/29/2023-11:09:50] [V] [TRT] Add_7 [Add] outputs: [output -> (3, 38365)[FLOAT]],  
[03/29/2023-11:09:50] [V] [TRT] Marking output_0 as output: output
[03/29/2023-11:09:50] [I] Finished parsing network model. Parse time: 0.0317031
Segmentation fault (core dumped)

# no verbose, with: --shapes=R:3x3,offset:3x1,alpha_shp:40x1,alpha_exp:10x1
[03/29/2023-11:14:26] [I] Start parsing network model.
[03/29/2023-11:14:26] [I] [TRT] ----------------------------------------------------------------
[03/29/2023-11:14:26] [I] [TRT] Input filename:   aaaa.onnx
[03/29/2023-11:14:26] [I] [TRT] ONNX IR version:  0.0.6
[03/29/2023-11:14:26] [I] [TRT] Opset version:    11
[03/29/2023-11:14:26] [I] [TRT] Producer name:    pytorch
[03/29/2023-11:14:26] [I] [TRT] Producer version: 1.12.1
[03/29/2023-11:14:26] [I] [TRT] Domain:            
[03/29/2023-11:14:26] [I] [TRT] Model version:    0
[03/29/2023-11:14:26] [I] [TRT] Doc string:        
[03/29/2023-11:14:26] [I] [TRT] ----------------------------------------------------------------
[03/29/2023-11:14:26] [I] [TRT] MatMul_2: broadcasting input1 to make tensors conform, dims(input0)=[3,38365,50][NONE] dims(input1)=[1,50,1][NONE].
[03/29/2023-11:14:26] [I] Finished parsing network model. Parse time: 0.0326259
[03/29/2023-11:14:26] [E] Static model does not take explicit shapes since the shape of inference tensors will be determined by the model itself
[03/29/2023-11:14:26] [E] Network And Config setup failed
[03/29/2023-11:14:26] [E] Building engine failed
[03/29/2023-11:14:26] [E] Failed to create engine from model or file.
[03/29/2023-11:14:26] [E] Engine set up failed
2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

Hello,

The onnx model is already attached, and the onnx checker passes okay. It is not possible to run the onnx from trtexec without converting it first.

5

Hi @elight1 ,

Would you like to check the relevant post.
This may help .

Thanks