TensorRT Inference form a .etlt model on Python

Andres_Godoy_Vanguard · October 26, 2021, 12:46am

Description

Hi, I followed the example on GitHub - NVIDIA/object-detection-tensorrt-example: Running object detection on a webcam feed using TensorRT on NVIDIA GPUs in Python., using a .engine finetuned from the [Nvidia's Catalog lpdNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:lpdnet). When inference is done there are two arrays as output one of length 4800 and the other 1200 which make no sense on the desired output, namely the bounding box coordinates. Using the mapping dictionary (Model Layout) the variables are still nonsense.

I’ve tried to use every SDK from Nvidia → TAO, DeepStream, TensorRT and the documentation is impossible! One spends more time solving issues on relatively easy tasks than developing applications, all for trying to use your hardware!

If I train a model on TAO, why is it so difficult to infer from it on python!

Environment

TensorRT Version: 7.1.3
CUDA Version: 10.2
CUDNN Version: 8.0
Operating System + Version: Jtpack 4.5.1
Python Version (if applicable): 3.8

NVES · October 26, 2021, 6:39am

Hi,
Can you try running your model with trtexec command, and share the “”–verbose"" log in case if the issue persist
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

You can refer below link for all the supported operators list, in case any operator is not supported you need to create a custom plugin to support that operation

github.com

onnx/onnx-tensorrt/blob/main/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 8.4 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT, clamping values to `+-INT_MAX` or `+-FLT_MAX` if necessary.

See below for the support matrix of ONNX operators in ONNX-TensorRT.

## Operator Support Matrix

| Operator                  | Supported  | Supported Types | Restrictions                                                                                                           |
|---------------------------|------------|-----------------|------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          | FP32, FP16, INT32 |
| Acos                      | Y          | FP32, FP16 |
| Acosh                     | Y          | FP32, FP16 |
| Add                       | Y          | FP32, FP16, INT32 |

This file has been truncated. show original

Also, request you to share your model and script if not shared already so that we can help you better.

Meanwhile, for some common errors and queries please refer to below link:

Thanks!

Andres_Godoy_Vanguard · October 26, 2021, 2:05pm

Hi, Thanks for the response…

The link you provided for trtexec is broken (404)

Operators in the model are okay because the model may be used with deepstream (I can’t use deepstream this time for this project, not a solution right now)

Attached is the project, just run the jupyter notebook and it should work… the outputs of length 4800 and 1200 are highlighted under the function len() afterward in the repo I linked in the initial post, they map those results with a model layout which I don’t know. Recap that the model is in the Nvidia’s Catalog as lpdNet.

spolisetty · October 27, 2021, 5:04pm

Hi,

Could you please confirm if you’re facing this error on using TAO(.etlt) or using only TensorRT.

Thank you.

Andres_Godoy_Vanguard · October 28, 2021, 12:23am

I re-trained the lpdNet on TAO and got the .etlt model, then made the conversion (tao-converter) into a .engine file.

That .engine is the one I want to run using python. If there is a way of running inference from the .etlt on python solves the issue as well.

I hope am clear Here.

What I need is to reproduce the notebook on the .zip file. and get bounding box coordinates.

spolisetty · October 28, 2021, 4:29am

Hi,

This looks like TAO related. Moving to TAO forum to get better help.

Morganh · October 28, 2021, 6:06am

It is possible to run inference with tensorrt engine generated by LPD model.

https://forums.developer.nvidia.com/t/issue-in-lpd-while-running-with-custom-python-script/191131/2

Officially, please see Integrating TAO CV Models with Triton Inference Server — TAO Toolkit 3.0 documentation and then leverage tao-toolkit-triton-apps/detectnet_processor.py at main · NVIDIA-AI-IOT/tao-toolkit-triton-apps · GitHub and tao-toolkit-triton-apps/utils.py at fc7e222c036354498e53a8ed11b5cf7c0a3e5239 · NVIDIA-AI-IOT/tao-toolkit-triton-apps · GitHub

Also, there are similar topics shared by other customers in TAO fourm.

system · November 16, 2021, 2:17am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
TAO and Jetson-Inference ...ooops TAO Toolkit jetson-inference	9	1045	February 20, 2023
Falure to do inference TAO Toolkit tensorrt	9	1070	January 11, 2022
Use pre-trained object detection TF2 models with TensorRT ONNX TensorRT	9	1904	May 31, 2021
How to use tlt trained model on Jetson Nano TAO Toolkit tensorrt , jetson-inference	7	2088	October 12, 2021
Inference error while using tensorrt engine on jetson nano Jetson Nano tensorrt , nvbugs	23	3530	April 20, 2022
YOLOv4 TensorRT inference results wayy off, but onnxruntime is not TensorRT tensorrt	7	908	June 7, 2022
Question regarding Tensorrt engine build vs inference environment (TensorRT version, Platform, etc) TensorRT	4	898	October 21, 2021
TensorRT Inference error on Jetson nano TensorRT	3	1183	December 6, 2021
TensorRT inference out of memory error TensorRT	1	2225	September 14, 2021
Using tensorrt .plan engine for object detection TensorRT	4	1113	January 19, 2022

TensorRT Inference form a .etlt model on Python

Description

Environment

Related topics