Trtexec : Error[4]: [graphShapeAnalyzer.cpp::analyzeShapes::1294] Error Code 4: Miscellaneous

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Description

Hello, I am using yolov5s.onnx ONNX model to run inference using trtexec CLI but I am facing an error (see below) when changing batch size.

image
image1344Ă—71 12.5 KB

Environment

TensorRT Version: trtexec command line interface
GPU Type: JEtson AGX ORIN
Nvidia Driver Version:
CUDA Version: 11.4
CUDNN Version: 8.3.2.49
Operating System + Version: ubuntu 20.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): baremetal

Relevant Files

Yolov5s ONNX model :
D_yolov5s.onnx (28.0 MB)

Steps To Reproduce

When you see the model on netron here you can see that it has Nx3x640x640 as input shapes and this means that you have a dynamic batch size because it has N So when running it using trtexec and change the batch size it will give me an error.

COMMAND:

  • $ /usr/src/tensorrt/bin/trtexec --onnx=D_yolov5s.onnx --int8 --explicitBatch --shapes=images:2x3x640x640 --verbose

here for example I put 2 as batch size but you can try using like 8, 16 or 32 etc…

NOTE:

I tested with batch equal to 1 and it worked, only with batch higher than one it will give me an error.

Thank you in advance.

Best regard,
Harry

2

Hi, Please refer to the below links to perform inference in INT8

Thanks!

4

Hello @NVES ,

Thank you for your reply. But even when using FP32 using this command below:

/usr/src/tensorrt/bin/trtexec --onnx=D_yolov5s.onnx --explicitBatch --shapes=images:2x3x640x640 --verbose

you will have the same error, please try to reproduce it by running the command on your Jetson AGX Orin and you will see the problem clearly.

Thank you.

Best regard,
Harry

5

Hi,

Looks like input node “images” do not have dynamic shape input(it’s defined as static input), that’s why it is working fine with batch size 1.
Please generate the ONNX model with dynamic shape input. You can also modify the ONNX model.

Thank you.

1 Like
6

Hello @spolisetty ,

Thank you for your answer, if you look on netron I modified the ONNX model into dynamic shapes so input node “images” support Nx3x640x640 so N is a dynamic batch size. So normaly it should works.

Thank you.

Best regard,
Harry

7

Hi,

It’s because the Reshape op has hard-coded shapes [1, 3, 85, 20, 20], which should have been [-1, 3, 85, 20, 20].

image
image1126Ă—576 26.1 KB

Could please let us know how you exported the ONNX model from PyT/TF? Do you use the dynamic_axes argument as in (optional) Exporting a Model from PyTorch to ONNX and Running it using ONNX Runtime — PyTorch Tutorials 1.12.1+cu102 documentation?

If you already did it, then you need to modify the Reshape of the PyT code from [1, 3, 85, 20, 20] to [-1, 3, 85, 20, 20].

Thank you.

1 Like
8

Hello @spolisetty ,

Thank you for your response, I used an other methode I hard coded the input shapes in to Nx3x640x640 wich apparently is not the right methode to do it.

So I have to try two other methodes:

  1. I will use this GiHub repo to download the ONNX model from pytorch using the script export.py you can add the flag --dynamic but when adding this option you will have indeed a dynamic ONNX model when verifiying in netron. But I had other errors in trtexec, I will share with you the error log as soon as possible after I run it on the Jetson AGX ORIN using trtexec.

  2. I will change the Reshape to [-1, 3, 85, 20, 20] as you suggested to me.

Thank you very much for your help, I will keep you updated.

Best regard,
Harry

9

Hello @spolisetty ,

This is my dynamic yolov5s ONNX model below:
yolov5s.onnx (27.6 MB)

when I run it using trtexec as before I have this error:

image
image1298Ă—307 31.2 KB

I used the GitHub repo here and add the --dynamic option to get the ONNX model in dynamic shapes, I verified the model on netron as well it is indeed dynamic shapes, you can verified as well.

Thank you in advance.

Best regard,
Harry

10

Hi @Harry-S,

We couldn’t reproduce the same error. We could successfully build the TensorRT engine.

[07/01/2022-10:00:49] [I]
&&&& PASSED TensorRT.trtexec [TensorRT v8401] # /opt/tensorrt/bin/trtexec --onnx=yolov5s.onnx --shapes=images:2x3x640x640--verbose

We recommend you to please use the latest version of the TensorRT 8.4 GA.

Thank you.

1 Like
11

Hello @spolisetty ,

Thank you very much for your reply.

I am using indeed TensorRT 8.4 see in the photo below.

image
image740Ă—139 10.2 KB

12

Hi,

We have tried on TensorRT version: 8.4.1.
Maybe you’re still using the 8.4 EA version, please upgrade your TensorRT version to 8.4 GA.
https://developer.nvidia.com/nvidia-tensorrt-8x-download

Also please try increasing the workspace size as some tactics need more workspace memory to run.

Thank you.

1 Like
13

Hello @spolisetty ,

I updated the TensorRT as you suggested to me and it worked see photo below:

image
image847Ă—241 19.4 KB

However, I am facing a new problem that CUDA is not installed, see below:

image
image1350Ă—122 6.73 KB

But CUDA is indeed installed see below with nvcc -V :

image

NOTE :

  • I update the system as well as suggested after installing it using debian package here and finaly ran this command : $ sudo apt-get update sudo apt-get install tensorrt libcudnn8
  • Do I need to install CUDA from here I think it is the mix of version between cuDNN, CUDA and TensorRT because it was working fine before updating TensorRT to the latest version and I am using JetPack 5.0 DP see below the version of JetPack from this command : $ sudo apt-cache show nvidia-jetpack

image

EDIT:

Actually I am using JetPack 5.0.1 apprently:

image
image1471Ă—536 37.3 KB

Thank you in advance.

Best regard,
Harry

14

Hi,

This looks like setup related issue on the Jetson.
We recommend you to please open a new post regarding setup issue on Jetson related forum to get better help.

Thank you.

15

Okay, thank you I will do it and put a link here so people can see because it was working fine before updating the trtexec.

Harry

EDIT:

here is the link to the new topic : CUDA is not working after updating trtexec

1 Like