Non-deterministic TensorRT engine building

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I got a detector model from Quantization Aware Training. Trained in Tensorflow 2.3, converted to onnx and then to TensorRT.
I noticed that when triggering several times a builder of a TRT engine from using the same code & variables, and applying inference on the same image from each TRT engine - the results are not the same. (the engine is built on the same device)
Does it mean that the TRT engine builder is non-deterministic?
If so, how can we make it be deterministic so that when triggering the same setup several times, we’ll get the same TRT engine?

Environment

TensorRT Version : 7.1.2
CUDA Version : 11.0
Operating System + Version : Ubuntu 18.04
Python Version (if applicable) : 3.6
TensorFlow Version (if applicable) : The model was trained on tf 2.3, converted to onnx, and then converted to tensorRT engine.

I can’t share the relevant model for this.

Any help will be appreciated!

3

Hi @weissrael,

Could you please provide the more information, the results are not the same means specifically? Is it just numeric precision which is accepted (e.g. difference like 10e-5 ) or obvious impact to the application (e.g. completely wrong bounding box)?

Thank you.

4

Hi,
It’s not numeric precision difference- we’re talking about completely wrong bounding box. In the comparisons I did, it’s the class of the bounding boxes.

5

Hi @weissrael,

TRT should be deterministic. We need to reproduce this issue to debug.
Could you please share us issue reproducible scripts and model.

Thank you.