Trtexec : Error[4]: [graphShapeAnalyzer.cpp::analyzeShapes::1294] Error Code 4: Miscellaneous

Description

Hello, I am using yolov5s.onnx ONNX model to run inference using trtexec CLI but I am facing an error (see below) when changing batch size.

Environment

TensorRT Version: trtexec command line interface
GPU Type: JEtson AGX ORIN
Nvidia Driver Version:
CUDA Version: 11.4
CUDNN Version: 8.3.2.49
Operating System + Version: ubuntu 20.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): baremetal

Relevant Files

Yolov5s ONNX model :
D_yolov5s.onnx (28.0 MB)

Steps To Reproduce

When you see the model on netron here you can see that it has Nx3x640x640 as input shapes and this means that you have a dynamic batch size because it has N So when running it using trtexec and change the batch size it will give me an error.

COMMAND:

  • $ /usr/src/tensorrt/bin/trtexec --onnx=D_yolov5s.onnx --int8 --explicitBatch --shapes=images:2x3x640x640 --verbose

here for example I put 2 as batch size but you can try using like 8, 16 or 32 etc…

NOTE:

I tested with batch equal to 1 and it worked, only with batch higher than one it will give me an error.

Thank you in advance.

Best regard,
Harry

Hi, Please refer to the below links to perform inference in INT8

Thanks!

Hello @NVES ,

Thank you for your reply. But even when using FP32 using this command below:

/usr/src/tensorrt/bin/trtexec --onnx=D_yolov5s.onnx --explicitBatch --shapes=images:2x3x640x640 --verbose

you will have the same error, please try to reproduce it by running the command on your Jetson AGX Orin and you will see the problem clearly.

Thank you.

Best regard,
Harry

Hi,

Looks like input node “images” do not have dynamic shape input(it’s defined as static input), that’s why it is working fine with batch size 1.
Please generate the ONNX model with dynamic shape input. You can also modify the ONNX model.

Thank you.

1 Like

Hello @spolisetty ,

Thank you for your answer, if you look on netron I modified the ONNX model into dynamic shapes so input node “images” support Nx3x640x640 so N is a dynamic batch size. So normaly it should works.

Thank you.

Best regard,
Harry

Hi,

It’s because the Reshape op has hard-coded shapes [1, 3, 85, 20, 20], which should have been [-1, 3, 85, 20, 20].

Could please let us know how you exported the ONNX model from PyT/TF? Do you use the dynamic_axes argument as in (optional) Exporting a Model from PyTorch to ONNX and Running it using ONNX Runtime — PyTorch Tutorials 1.11.0+cu102 documentation?

If you already did it, then you need to modify the Reshape of the PyT code from [1, 3, 85, 20, 20] to [-1, 3, 85, 20, 20].

Thank you.

1 Like

Hello @spolisetty ,

Thank you for your response, I used an other methode I hard coded the input shapes in to Nx3x640x640 wich apparently is not the right methode to do it.

So I have to try two other methodes:

  1. I will use this GiHub repo to download the ONNX model from pytorch using the script export.py you can add the flag --dynamic but when adding this option you will have indeed a dynamic ONNX model when verifiying in netron. But I had other errors in trtexec, I will share with you the error log as soon as possible after I run it on the Jetson AGX ORIN using trtexec.

  2. I will change the Reshape to [-1, 3, 85, 20, 20] as you suggested to me.

Thank you very much for your help, I will keep you updated.

Best regard,
Harry

Hello @spolisetty ,

This is my dynamic yolov5s ONNX model below:
yolov5s.onnx (27.6 MB)

when I run it using trtexec as before I have this error:

I used the GitHub repo here and add the --dynamic option to get the ONNX model in dynamic shapes, I verified the model on netron as well it is indeed dynamic shapes, you can verified as well.

Thank you in advance.

Best regard,
Harry

Hi @Harry-S,

We couldn’t reproduce the same error. We could successfully build the TensorRT engine.

[07/01/2022-10:00:49] [I]
&&&& PASSED TensorRT.trtexec [TensorRT v8401] # /opt/tensorrt/bin/trtexec --onnx=yolov5s.onnx --shapes=images:2x3x640x640--verbose

We recommend you to please use the latest version of the TensorRT 8.4 GA.

Thank you.

1 Like

Hello @spolisetty ,

Thank you very much for your reply.

I am using indeed TensorRT 8.4 see in the photo below.