Convert int8-onnx model to trt engine?

Chubby · February 16, 2023, 9:29am

I tried to convert model.int8.onnx to tensorRT engine, but encountered error…

The model was already quantized in onnxRT framework and it has Quantized/Dequantized linear layer, whuch are supported by tensorRT accoring to this page (onnx-tensorrt/operators.md at main · onnx/onnx-tensorrt · GitHub)

I used command:
./trtexec --onnx=test.int8-onnx-calibrated.onnx --saveEngine=engine.trt

I thoght that it could be converted … But errors appeared, which are following.

It’s not possible to convert int8-onnx model to trt engine?
Best regards.

[02/16/2023-18:18:25] [I] [TRT] ----------------------------------------------------------------
[02/16/2023-18:18:25] [I] [TRT] Input filename:   test.int8-onnx-calibrated.onnx
[02/16/2023-18:18:25] [I] [TRT] ONNX IR version:  0.0.6
[02/16/2023-18:18:25] [I] [TRT] Opset version:    11
[02/16/2023-18:18:25] [I] [TRT] Producer name:    onnx.utils.extract_model
[02/16/2023-18:18:25] [I] [TRT] Producer version:
[02/16/2023-18:18:25] [I] [TRT] Domain:
[02/16/2023-18:18:25] [I] [TRT] Model version:    0
[02/16/2023-18:18:25] [I] [TRT] Doc string:
[02/16/2023-18:18:25] [I] [TRT] ----------------------------------------------------------------
[02/16/2023-18:18:25] [W] [TRT] onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.

[02/16/2023-18:18:25] [E] Error[2]: [scaleNode.cpp::getChannelAxis::27] Error Code 2: Internal Error (features.0.bias_DequantizeLinear_quantize_scale_node: out of bounds channel axis 1. Number of input dimensions is 1.)
[02/16/2023-18:18:25] [E] [TRT] ModelImporter.cpp:720: While parsing node number 0 [DequantizeLinear -> "features.0.bias"]:
[02/16/2023-18:18:25] [E] [TRT] ModelImporter.cpp:721: --- Begin node ---
[02/16/2023-18:18:25] [E] [TRT] ModelImporter.cpp:722: input: "features.0.bias_quantized"
input: "features.0.bias_quantized_scale"
input: "features.0.bias_quantized_zero_point"
output: "features.0.bias"
name: "features.0.bias_DequantizeLinear"
op_type: "DequantizeLinear"

[02/16/2023-18:18:25] [E] [TRT] ModelImporter.cpp:723: --- End node ---
[02/16/2023-18:18:25] [E] [TRT] ModelImporter.cpp:726: ERROR: ModelImporter.cpp:179 In function parseGraph:
[6] Invalid Node - features.0.bias_DequantizeLinear
[scaleNode.cpp::getChannelAxis::27] Error Code 2: Internal Error (features.0.bias_DequantizeLinear_quantize_scale_node: out of bounds channel axis 1. Number of input dimensions is 1.)

[02/16/2023-18:18:25] [E] Failed to parse onnx file
[02/16/2023-18:18:25] [I] Finish parsing network model
[02/16/2023-18:18:25] [E] Parsing model failed
[02/16/2023-18:18:25] [E] Failed to create engine from model.
[02/16/2023-18:18:25] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8205] # ./trtexec --onnx=test.int8-onnx-calibrated.onnx

A clear and concise description of the bug or issue.

Environment

TensorRT Version:
GPU Type:
Nvidia Driver Version:
CUDA Version:
CUDNN Version:
Operating System + Version:
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

spolisetty · February 16, 2023, 9:47am

Hi,

We recommend that you try the latest TensorRT version 8.5.3 and please share with us complete verbose logs and issue repro ONNX model if you still face the issue.

Thank you.

AnotherChubby · February 17, 2023, 12:57am

I think it also encountered the same issue

[02/17/2023-09:54:47] [I] TensorRT version: 8.5.3
[02/17/2023-09:54:53] [I] [TRT] [MemUsageChange] Init CUDA: CPU +533, GPU +0, now: CPU 540, GPU 1282 (MiB)
[02/17/2023-09:54:53] [I] Start parsing network model
[02/17/2023-09:54:53] [I] [TRT] ----------------------------------------------------------------
[02/17/2023-09:54:53] [I] [TRT] Input filename:   test.int8-onnx-calibrated.onnx
[02/17/2023-09:54:53] [I] [TRT] ONNX IR version:  0.0.6
[02/17/2023-09:54:53] [I] [TRT] Opset version:    11
[02/17/2023-09:54:53] [I] [TRT] Producer name:    onnx.utils.extract_model
[02/17/2023-09:54:53] [I] [TRT] Producer version:
[02/17/2023-09:54:53] [I] [TRT] Domain:
[02/17/2023-09:54:53] [I] [TRT] Model version:    0
[02/17/2023-09:54:53] [I] [TRT] Doc string:
[02/17/2023-09:54:53] [I] [TRT] ----------------------------------------------------------------
[02/17/2023-09:54:53] [W] [TRT] onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[02/17/2023-09:54:53] [E] Error[2]: [scaleNode.cpp::getChannelAxis::27] Error Code 2: Internal Error (features.0.bias_DequantizeLinear_quantize_scale_node: out of bounds channel axis 1. Number of input dimensions is 1.)
[02/17/2023-09:54:53] [E] [TRT] ModelImporter.cpp:720: While parsing node number 0 [DequantizeLinear -> "features.0.bias"]:
[02/17/2023-09:54:53] [E] [TRT] ModelImporter.cpp:721: --- Begin node ---
[02/17/2023-09:54:53] [E] [TRT] ModelImporter.cpp:722: input: "features.0.bias_quantized"
input: "features.0.bias_quantized_scale"
input: "features.0.bias_quantized_zero_point"
output: "features.0.bias"
name: "features.0.bias_DequantizeLinear"
op_type: "DequantizeLinear"

[02/17/2023-09:54:53] [E] [TRT] ModelImporter.cpp:723: --- End node ---
[02/17/2023-09:54:53] [E] [TRT] ModelImporter.cpp:726: ERROR: ModelImporter.cpp:179 In function parseGraph:
[6] Invalid Node - features.0.bias_DequantizeLinear
[scaleNode.cpp::getChannelAxis::27] Error Code 2: Internal Error (features.0.bias_DequantizeLinear_quantize_scale_node: out of bounds channel axis 1. Number of input dimensions is 1.)
[02/17/2023-09:54:53] [E] Failed to parse onnx file
[02/17/2023-09:54:53] [I] Finish parsing network model
[02/17/2023-09:54:53] [E] Parsing model failed
[02/17/2023-09:54:53] [E] Failed to create engine from model or file.
[02/17/2023-09:54:53] [E] Engine set up failed

spolisetty · February 17, 2023, 5:34am

Please share the issue repro ONNX model with us here or via DM for better debugging.

Thank you.

AnotherChubby · February 17, 2023, 7:16am

The suspicious reason is that conv has three input operators: input, weghit, bias in my onnx model.

However, trtexec assumes two input channels: input, weight+bias(?). When using TensorRT/tools/pytorch-quantization at main · NVIDIA/TensorRT · GitHub and quantize the model, a generated onnx model has 2 input and it seems to be possible to convert to trt.engine via trtexec.

No support of convirsion of “onnx-int8” model to trt.engine?

コメント 2023-02-17 160807

AnotherChubby · February 21, 2023, 11:11am

No methods?