To try and push for a solution, I have a step-by-step reproducible example, so that interested parties can identify the issue.
We start with a Tensorflow
saved_model.pb, and export it to run in TensorRT. We download an official Tensorflow model, and export with
tf2onnx.convert. See the steps here:
$ wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8.tar.gz $ tar xvfz ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8.tar.gz $ python3 -m tf2onnx.convert --saved-model ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/saved_model/ --opset 11 --output ssd_model.onnx
To run inference, I adapt very slightly the ONNX ResNet50 example found at
All of the files are available at this gist I’m providing. For this next step we’ll need the
onnx_to_tensorrt.py file, as well as
$ python3 onnx_to_tensorrt.py 'ssd_model.onnx'. You’ll see the error:
Unsupported ONNX data type: UINT8.
If we are to look at this official advice from NVidia, we can sidestep this issue by running the
python3 graph_surgeon.py (this file is also in the gist). Ensure you install graph surgeon according to the linked instructions. This should apparently fix our incorrect data types.
We can verify this by running again with our updated model, parsed with Graph Surgeon.
python3 onnx_to_tensorrt.py 'updated_ssd_model.onnx'.
However, it seems that the Graph Surgeon approach didn’t completely solve our issue, and you will see a new error:
Beginning ONNX file parsing [TensorRT] WARNING: onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped [TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin NonMaxSuppression version 1
This error also occurs with other proprietary models I have tried. But these publicly available models show the same behaviour.
I don’t have any solution beyond this point. I have tried converting my model to UFF, and have failed, and I have tried converting my model to ONNX, and have failed in this path. Anyone have any thoughts?