TensorRT inference produces unexpected results

I am adding support for TensorRT models in an open source project called DonkeyCar which is a self-driving RC Car.

Here is the source code that freezes the model.

Here is the code that does the inference
https://github.com/autorope/donkeycar/blob/dev/donkeycar/parts/tensorrt.py

When testing the converted UFF model I see TensorRT perform well with some sets of input data. Here is an example where the model does well:

Pasteboard - Uploaded Image [Good TensorRT results]

However sometimes it produces very unexpected results, while the original Keras based (.h5) that the UFF model is based on performs well.

Pasteboard - Uploaded Image [Good fit based on Keras Model]
Pasteboard - Uploaded Image [Bad TensorRT UFF Model

The second graph shows that inference is quite poor. The converted model is based on the Keras model shown in the above graph.

I am using TF 1.13.1 (with TF-GPU)
CUDA 10.0
TensorRT 5.1 GA


Do you any thoughts on how I can fix this ?

Did not realize that this was a place for issues with the container.

x-post at https://devtalk.nvidia.com/default/topic/1057157/tensorrt/tensorrt-inference-produces-unexpected-results/