Reproducible step-by-step ONNX to TensorRT issue: Unsupported ONNX data type: UINT8

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
, ,
1

Aspects of this issue have been seen here, here, here, and here, but there is not a clear solution.

To try and push for a solution, I have a step-by-step reproducible example, so that interested parties can identify the issue.

We start with a Tensorflow saved_model.pb, and export it to run in TensorRT. We download an official Tensorflow model, and export with tf2onnx.convert. See the steps here:

$ wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8.tar.gz
$ tar xvfz ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8.tar.gz
$ python3 -m tf2onnx.convert --saved-model ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/saved_model/ --opset 11 --output ssd_model.onnx

To run inference, I adapt very slightly the ONNX ResNet50 example found at /usr/src/tensorrt/samples/python/introductory_parser_samples/onnx_resnet50.py

All of the files are available at this gist I’m providing. For this next step we’ll need the onnx_to_tensorrt.py file, as well as common.py, coco_labels.txt and data_processing.py.

Run $ python3 onnx_to_tensorrt.py 'ssd_model.onnx'. You’ll see the error: Unsupported ONNX data type: UINT8.

If we are to look at this official advice from NVidia, we can sidestep this issue by running the python3 graph_surgeon.py (this file is also in the gist). Ensure you install graph surgeon according to the linked instructions. This should apparently fix our incorrect data types.

We can verify this by running again with our updated model, parsed with Graph Surgeon.

Run: python3 onnx_to_tensorrt.py 'updated_ssd_model.onnx'.

However, it seems that the Graph Surgeon approach didn’t completely solve our issue, and you will see a new error:

Beginning ONNX file parsing
[TensorRT] WARNING: onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin NonMaxSuppression version 1

This error also occurs with other proprietary models I have tried. But these publicly available models show the same behaviour.

I don’t have any solution beyond this point. I have tried converting my model to UFF, and have failed, and I have tried converting my model to ONNX, and have failed in this path. Anyone have any thoughts?

3

Hi,

[TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin NonMaxSuppression version 1

This error indicates TensorRT doesn’t support the NonMaxSuppression operation, which is used in your TensorFlow model.
So you will need to add the support to TensorRT with our plugin API below:
https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_plugin.html

Thanks.

4

Thanks for the help! I’m not that familiar with TensorRT yet, so didn’t realise that meant there was an unsupported operation.

I guess TF-TRT is probably the way to go here, if I don’t have the development budget to add the operation?

I think I’ve seen some info about that having some performance penalties relative to pure TensorRT? Is there any data or info on that?

5

Hi,

If the plugin is not an option, you can give TF-TRT a try.
But please noted that TensorRT supported layer in TF-TRT is different from the standalone TensorRT.

You can find the list in the below document:

Thanks.