TensorRT inference produces unexpected results

I am adding support for TensorRT models in an open source project called DonkeyCar which is a self-driving RC Car.

Here is the source code that freezes the model.

Here is the code that does the inference

When testing the converted UFF model I see TensorRT perform well with some sets of input data. Here is an example where the model does well:

https://pasteboard.co/Inuz69S.png [Good TensorRT results]

However sometimes it produces very unexpected results, while the original Keras based (.h5) that the UFF model is based on performs well.

https://pasteboard.co/InuAY8O.png [Good fit based on Keras Model]
https://pasteboard.co/InuBxXa.png [Bad TensorRT UFF Model

The second graph shows that inference is quite poor. The converted model is based on the Keras model shown in the above graph.

I am using TF 1.13.1 (with TF-GPU)
CUDA 10.0
TensorRT 5.1 GA

Do you any thoughts on how I can fix this ?

Did not realize that this was a place for issues with the container.

x-post at https://devtalk.nvidia.com/default/topic/1057157/tensorrt/tensorrt-inference-produces-unexpected-results/