Use pre-trained object detection TF2 models with TensorRT ONNX

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Hi,

I’m trying to use a pre-trained object detection model from TF2 Model Zoo with TensorRT, but I’m stucked due to errors in engine building.
In particular I would like to convert TF2 saved_model.pb in ONNX format, optimize it with TensorRT and perfom inference with TensorRT engine.

I followed these steps:

  1. Download model from TF2 Model Zoo: Faster R-CNN ResNet50 V1 640x640

  2. Convert saved_model.pb to ONNX whit this command:
    python3 -m tf2onnx.convert --saved-model resnet/saved_model/ --opset 13 --output resnet.onnx

  3. Check ONNX file with this code:
    import onnx
    try:
     onnx_model = onnx.load(model_path)
     onnx.checker.check_model(onnx_model)
     print("Model is valid")
    except Exception as e:
     print(e)

  4. Use buildEngine.py for building TRT engine. I used code posted here NVIDIA Tutorial (Tensorflow 2 code example).
    I run the command:
    python3 buildEngine.py --onnx_file resnet.onnx

At first run I got this error:
Unsupported ONNX data type: UINT8

Then I used solution proposed here.

So i run again:
python3 buildEngine.py --onnx_file resnet_f32.onnx

and now I have this error:

[TensorRT] WARNING: onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
Traceback (most recent call last):
File “buildEngine.py”, line 31, in
main(args)
File “buildEngine.py”, line 22, in main
engine = eng.build_engine(onnx_path, shape=shape)
File “engine.py”, line 16, in build_engine
parser.parse(model.read())
IndexError: Attribute not found: axes

So my questions:

  • How can I solve this?

  • It is possible to use TF2 object detection pre-trained models with TensorRT?

Thank you very much for support.

Environment

TensorRT Version: 7.2.2.3
GPU Type: NVIDIA GeForce RTX 2070 SUPER, 8GB
Nvidia Driver Version: 460.73.01
CUDA Version: 11.0.3
CUDNN Version: 8.0.5.39
Operating System + Version: Ubuntu 20.04.2 LTS
Python Version (if applicable): 3.8
ONNX Version: 1.9.0
ONNX-GraphSurgeon Version: 0.2.6
TF2ONNX Version: 1.8.4
Baremetal or Container (if container which image + tag): Baremetal

1 Like
5

Hi @cint.lorenzo,

We request you to share issue reproducible ONNX model and script to try from our end for better assistance.
Meanwhile we also recommend you to try using trtexec.
For your reference,
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

Thank you.

1 Like
6

Hi @spolisetty and thanks for reply.

Here you can download the ONNX model:
https://we.tl/t-mekUpBBN8N

Here is the script from NVIDIA tutorial:
buildEngine.py (897 Bytes)

You can run it with:
python3 buildEngine.py --onnx_file resnet50.onnx

I also tried with trtexec, with this command:
trtexec --onnx=resnet50.onnx
and I got the same error.

Here the trtexec log:
trtexec_log.txt (3.9 KB)

Thank you very much.

7

Hi @cint.lorenzo,

We could reproduce the same error. Looks like you are using the opset 13 version, which is currently unsupported for TRT 7.2.2.3. Can you try exporting your model to a lower opset (i.e. opset 11).

We also recommend you to checkout onnx-simplifier.

Thank you.

1 Like
8

Hi @spolisetty and thanks again for your support.

I tried to export the model with opset 11 and now I get these errors when running buildEngine.py:

[TensorRT] WARNING: onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] ERROR: Network must have at least one output
[TensorRT] ERROR: Network validation failed.

I tried also onnx-semplifier, but also there I got an error:

onnx.onnx_cpp2py_export.checker.ValidationError: Nodes in a graph must be topologically sorted, however input ‘__inference_Preprocessor_ResizeToRange_cond_false_13101_567_const_zero__42:0’ of node:
name: OpType: Slice is not output of any previous nodes.

Thank you very much.

9

Hi @cint.lorenzo,

Could you please share with us new ONNX model(generated with opset 11) to try from our end for better assistance.

Thank you.

10

Hi @spolisetty,

here you can find ONNX model generated with opset 11:
https://we.tl/t-ZAiEyftqDD

Thank you again for your support.

11

Hi @cint.lorenzo,

Sorry for the delayed response, we recommend you to try Polygraphy. We also recommend you to use latest TensorRT version 8.0 EA.
Please run polygraphy surgeon sanitize --fold-constants model.onnx -o folded.onnx before running trtexec.
Let us know if you still face this issue.

Thank you.

1 Like
12

Hi @spolisetty and thank you again for your support.

Unfortunately I still got errors.
I downloaded TensoRT version 8.0 EA and applied Polygraphy as suggested.

This is the trtexec output on the folded model:
trtexec_log.txt (6.0 KB)

Here you can find the model:
folded_fix.onnx

These are the steps to reproduce:

  1. Original model: http://download.tensorflow.org/models/object_detection/tf2/20200711/faster_rcnn_resnet50_v1_1024x1024_coco17_tpu-8.tar.gz

  2. Conversion to ONNX with:
    python3 -m tf2onnx.convert --saved-model resnet/saved_model/ --opset 11 --output resnet.onnx

  3. Apply Polygraphy: polygraphy surgeon sanitize --fold-constants resnet.onnx -o folded.onnx

  4. If I run trtexec with model from point 3, I got this error:
    trtexec_log_uint.txt (4.5 KB)

  5. Use a script for format conversion from uint to float:
    fix_onnx_model.py (217 Bytes)

  6. Run on model from point 5: trtexec --onnx=folded_fix.onnx

  7. Get error attached at the start of the message. Here the verbose version:
    trtexec_log_verbose.txt (268.0 KB)

Hope there is all the information needed.

Thank you.

13

Hi @cint.lorenzo,

“NonMaxSuppression” is currently not supported in ONNX parser. We may support in future releases.
Please refer How to use NMS with Pytorch model (that was converted to ONNX -> TensorRT) · Issue #795 · NVIDIA/TensorRT · GitHub

You need to create a custom plugin for any unsupported layer in your model. Please refer to below samples:

Custom plugin for ONNX:
https://github.com/NVIDIA/TensorRT/issues/6#issuecomment-603683069

Thank you.