Padding and speedup of tensorrt inference

kirpasaccessory · August 24, 2021, 1:56pm

I have tensorflow model what I converted to ONXX model. As you can see on attached model sumerry image, models has convolution layers with padding 1,1,1,1 but my actual input image size is 256x256. So, can you tell me if pad 1 is killing my tensorrt performance??

NVES · August 24, 2021, 6:01pm

Hi,
Can you try running your model with trtexec command, and share the “”–verbose"" log in case if the issue persist
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

You can refer below link for all the supported operators list, in case any operator is not supported you need to create a custom plugin to support that operation

github.com

onnx/onnx-tensorrt/blob/main/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 8.4 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT, clamping values to `+-INT_MAX` or `+-FLT_MAX` if necessary.

See below for the support matrix of ONNX operators in ONNX-TensorRT.

## Operator Support Matrix

| Operator                  | Supported  | Supported Types | Restrictions                                                                                                           |
|---------------------------|------------|-----------------|------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          | FP32, FP16, INT32 |
| Acos                      | Y          | FP32, FP16 |
| Acosh                     | Y          | FP32, FP16 |
| Add                       | Y          | FP32, FP16, INT32 |

This file has been truncated. show original

Also, request you to share your model and script if not shared already so that we can help you better.

Meanwhile, for some common errors and queries please refer to below link:

Thanks!

Topic		Replies	Views
Tensorrt Execution Provider TensorRT tensorrt , cudnn , onnx	1	742	November 27, 2023
Detectron2: faster inferencing TensorRT	2	1383	April 29, 2022
TensorRT model giving constant output TensorRT deepstream	4	1340	November 30, 2021
ONNX Model Int64 Weights TensorRT	12	12814	February 17, 2024
Does TensorRT exploit parallelism in a computational graph during inference? TensorRT	2	524	April 18, 2023
Problem converting TensorFlow 2-> ONNX model to TensorRT Engine (efficientdet_d0) TensorRT	8	1383	November 17, 2022
Centernet model TensortRT file type conversion TensorRT tensorrt , tensorflow , onnx	1	781	November 2, 2022
Inference TensorRT randomly returns nan TensorRT tensorrt	2	516	April 27, 2023
Tensorrt Conversion from ONNX for keras-ocr models fails because of int32 input in intermediate layers TensorRT	1	733	March 8, 2023
Mod operator unsupported in TensorRT 8.4.1 (included w/ Jetpack 5.0.2) TensorRT jetpack , tensorrt , cuda , jetson-inference , onnx	5	1533	January 2, 2023

Padding and speedup of tensorrt inference

Related topics