Tensorrt Execution Provider

vudn · November 13, 2023, 8:30am

Description

I inference model by provider TensorrtExecutionProvider of ONNXRuntime.
First it convert model ONNX to Engine
Then it execute inference model

I have 3 questions:
Question 1: The inference time for first time is too slow. How can I fix it ???. As I know maybe I can save .profile and load it before inference.
Question 2: How can I save and load file .engine instead of reconverting to engine after each inference.
Question 3: I can not set trt_profile_max_shapes (it said not support) for inference. Because when I give a large input dataloader it raise error mylin and limit of shape.
Besides, Any support for ONNX IOBinding with TensorrtExecutionProvider ???

Thanks a lot for your help <3

Environment

TensorRT Version: 8.2.3
GPU Type: A10
Nvidia Driver Version: 11.4
CUDA Version: 11.6
CUDNN Version: 8.2
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): 3.8
PyTorch Version (if applicable): 1.8
ONNX Runtime GPU: 1.11.0

Relevant Files

build_model_onnx.zip (1.5 KB)

AakankshaS · November 27, 2023, 7:44am

Hi,
Can you try running your model with trtexec command, and share the “”–verbose"" log in case if the issue persist

You can refer below link for all the supported operators list, in case any operator is not supported you need to create a custom plugin to support that operation

github.com

onnx/onnx-tensorrt/blob/main/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 8.6 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT, clamping values to `+-INT_MAX` or `+-FLT_MAX` if necessary.

See below for the support matrix of ONNX operators in ONNX-TensorRT.

## Operator Support Matrix

| Operator                  | Supported  | Supported Types | Restrictions                                                                                                           |
|---------------------------|------------|-----------------|------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          | FP32, FP16, INT32 |
| Acos                      | Y          | FP32, FP16 |
| Acosh                     | Y          | FP32, FP16 |
| Add                       | Y          | FP32, FP16, INT32 |

This file has been truncated. show original

Also, request you to share your model and script if not shared already so that we can help you better.

Meanwhile, for some common errors and queries please refer to below link:

Thanks!