Deterministic TensorRT optimization

Description

When I optimize the same model with tensorRT_optimization tool on Nvidia Drive AGX platform twice in a row , I get two binary files of different sizes. Also, it is mentioned here https://github.com/NVIDIA/framework-determinism that TensorRT behaves non-deterministically. Drive OS comes with TensorRT 5.1.2. Is it fixed in newer versions? Can the model be optimized deterministically using TensorRT C++ API? When will solution be available on Nvidia Drive?

Environment

TensorRT Version: 5.1.2
CUDA Version: 10.2

Hi @domagoj.krivosic
If you are using same engine with same input, TensorRT should be deterministic.
Can you please try this on latest TRT release.

Thanks!

Sorry if I wasn’t clear enough. I am not comparing inference results, I am generating engines from UFF file. When I generate two engines from the same UFF file with the same arguments, the engine files have different sizes.

Hi @domagoj.krivosic,
Can you help me with the model and the script you are using.
Thanks!

Hi @AakankshaS ,

Unfortunately, we can’t share the model at this point (we might prepare a sample model a bit later). However, we don’t see this as a question specific to the model we are currently looking at. We plan to use TensorRT for optimization of various models in future, so at this point we are just wondering about the general behavior of the TensorRT product. If we optimize the same model (with the same parameters - weights) twice, should we expect to get exactly the same engine files? If not, is it possible to achieve this reproducibility?

We are working on the Nvidia Drive platform (Drive OS 5.1.6.1), optimizing with the shipped binary /usr/local/driveworks/tools/dnn/tensorRT_optimization, but the question is not limited to the mentioned tool. We would like to know if the reproducibility is achievable with TensorRT API or any other way.

Hi @domagoj.krivosic,

The below link will help you answer your query
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#deter-repro
https://docs.nvidia.com/deeplearning/tensorrt/sample-support-guide/index.html#samplealgorithmselection

Thanks!

@AakankshaS thank you. I see that algorithm selector is available from TensorRT version 7, but not in earlier versions. Is it safe to say that fully deterministic engine building is supported only with TensorRT 7.0 and newer?

Hi @domagoj.krivosic,
Yes, we can say that. With latest TRT Releases, you will find more features with improved performance.
Thanks!

1 Like