Failed to parse onnx file, Engine creation failed. INT64. Attempting to cast down to INT32

[yaduvir.singh]

Description

I am trying to convert a tensorflow model to onnx using tensorrt and Then using trtexec on the .onnx file throws int32 datatype error. How to solve this issue? TRT downcast INT64 weights to INT32 and then ONNX engine failed because it do not support INT32.How to resolve this issue.

Environment

TensorRT Version: 7.0.0.11
GPU Type: Tesla K80
Nvidia Driver Version: 440.64.00
CUDA Version: CUDA Version: 10.2
CUDNN Version: 7.6
Operating System + Version: Ubuntu x86_64
Python Version (if applicable): 3.6
TensorFlow Version (if applicable): 1.15.2
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): barematel

Relevant Files

Model files: saved_model, frozen_graph_int8.onnx

frozen_graph_int8.onnx

saved_model.zip

Steps To Reproduce

~$ python -m tf2onnx.convert --saved-model saved_model/1591286324/ --output frozen_graph_int8.onnx --opset 11

~$ trtexec --explicitBatch --onnx=frozen_graph_int8.onnx

Input filename: frozen_graph_int8.onnx
ONNX IR version: 0.0.6
Opset version: 11
Producer name: tf2onnx
Producer version: 1.5.6
Domain:
Model version: 0
Doc string:

[06/04/2020-16:03:36] [W] [TRT] /home/ubuntu/notebook/TensorRT/parsers/onnx/onnx2trt_utils.cpp:235: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[06/04/2020-16:03:36] [W] [TRT] /home/ubuntu/notebook/TensorRT/parsers/onnx/onnx2trt_utils.cpp:261: One or more weights outside the range of INT32 was clamped
While parsing node number 30 [MatMul]:
ERROR: /home/ubuntu/notebook/TensorRT/parsers/onnx/builtin_op_importers.cpp:2080 In function importMatMul:
[8] Assertion failed: inputA->getType() == inputB->getType() && inputA->getType() != nvinfer1::DataType::kINT32
[06/04/2020-16:03:36] [E] Failed to parse onnx file
[06/04/2020-16:03:36] [E] Parsing model failed
[06/04/2020-16:03:36] [E] Engine creation failed
[06/04/2020-16:03:36] [E] Engine set up failed

[SunilJB]

I tried both the save model file and getting following error using trtexec execution:
Unsupported ONNX data type: UINT8 (2)
ERROR: image_tensor:0:189 In function importInput:
[8] Assertion failed: convertDtype(onnxDtype.elem_type(), &trtDtype)

Issue seems to be due to incorrect dtype (UINT8) of image_tensor node.
Please refer below link for more details:

tensorrt inference error while load onnx model

Hi, TensorRT only supports FP32, FP16, INT32, and INT8: https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/infer/FoundationalTypes/DataType.html#tensorrt.DataType UINT8 is not supported as mentioned there. Can you try to use int8 or int32 instead of uint8 before converting your model to ONNX? Alternatively, you could try to use the ONNX API to convert the UINT8 nodes to INT8 or INT32 after training/converting to ONNX, but these could potentially create incorrect results if not h…

Thanks

[yaduvir.singh]

Thanks!. I have uploaded saved_model/1591286324_int8 in the original post and updated logs with onnx engine setup failed error.

[yaduvir.singh]

@SunilJB, Have you checked the new model for the errors, i have updated original post with new saved_model and onnx link.

[SunilJB]

Thanks for sharing the model files, we will look into it and update you accordingly.

Thanks

[yaduvir.singh]

@SunilJB Hi, is there any update?