Question about TensorRT reproducibility on different architectures

[nk_white]

Description

I have a general question about the expectation of TensorRT engine files.

I used Transfer Learning Toolkit/TAO Toolkit on a dGPU to train a custom YOLOv3 model. I iterated through several pruning and retraining steps until I had a model that worked really well on my training set and test sets. I then created a .etlt file and finally a TensorRT .engine file and ran inference on a test set on the dGPU and was happy with the results. Then, I moved the model to the Jetson Nano by creating a .engine file on the Nano, using the same tlt-converter command line as I did with the dGPU. However, the inference results - as measured by eye (bounding boxes in Deepstream 5.1) were much worse than on the dGPU. I don’t know of an easy way to just do inference on the Nano using a .engine file to test the results quantitatively. But my question is this: Should TensorRT engines produce the same results on different platforms? If so, then I will continue to try to figure out how to quantitatively measure inference on the TLT YOLOv3 model outside of Deepstream 5.1. If not, then how does one select the proper model on the dGPU to deploy to edge devices?

Environment

TensorRT Version: 7.2.1
GPU Type: Tesla V100
Nvidia Driver Version: 460.32.03
CUDA Version:
CUDNN Version:
Operating System + Version: CentOS7
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

• Exact steps/commands to build your repro
• Exact steps/commands to run your repro
• Full traceback of errors encountered

[spolisetty]

Hi @nk_white,

TensorRT engines do not produce the same results on different platforms. Please refer following post to know more details on your query.

Is TensorRT inference deterministic/reproducibile?

Hi @daniel.widmann, If you are using same engine with same input, TensorRT should be deterministic. However I don’t think engine building is supposed to be deterministic as tactics are chosen based on observed runtime. If you’re outputting your log with info level, you should be able to compare tactic selection between the two engines. Since different tactics/kernels could change order of operations, you would expect floating point differences. You can refer to the below link. Thanks!

Thank you.

[nk_white]

Thank you for your reply. From the explanation, I interpret “floating point differences” to be qualitatively equal. The prediction performance I’m seeing is not qualitatively equal so I will continue on in an effort to figure out why my model is performing so poorly in DeepStream 5.1.