I am adding support for TensorRT models in an open source project called DonkeyCar which is a self-driving RC Car.
Here is the source code that freezes the model.
Here is the code that does the inference
https://github.com/autorope/donkeycar/blob/dev/donkeycar/parts/tensorrt.py
When testing the converted UFF model I see TensorRT perform well with some sets of input data. Here is an example where the model does well:
Pasteboard - Uploaded Image [Good TensorRT results]
However sometimes it produces very unexpected results, while the original Keras based (.h5) that the UFF model is based on performs well.
Pasteboard - Uploaded Image [Good fit based on Keras Model]
Pasteboard - Uploaded Image [Bad TensorRT UFF Model
The second graph shows that inference is quite poor. The converted model is based on the Keras model shown in the above graph.
I am using TF 1.13.1 (with TF-GPU)
CUDA 10.0
TensorRT 5.1 GA
Do you any thoughts on how I can fix this ?