Int8 inference result is differ between 2080ti and Xavier

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I used the same int8 calibration to build engine in 2080ti and Xavier, 2080ti’s version is tensorrt8, Xavier’s is tensorrt7, I’m sure the two versions calibration have the same key-value（layer:scale）,but result of two HW platforms differ greatly，The accuracy of 2080ti’s result is higher than that of Xavier, how to solve this problem? thanks

Environment

TensorRT Version:
GPU Type:
Nvidia Driver Version:
CUDA Version:
CUDNN Version:
Operating System + Version:
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2

Hi,

We recommend you to please use the latest TensorRT version on Xavier to get the best performance, there are known issues resolved in the latest version.

Thank you.

3

Hi, Please refer to the below links to perform inference in INT8

Thanks!

6

Thank you for your reply. I have another question, if different version of trt’s int8 calibration has the same key-value(layer name : scale ), does it means the INT8 calibration table can be compatible, and the results are similar？

7

Thank you for your reply. I have another question, if different version of trt’s int8 calibration has the same key-value(layer name : scale ), does it means the INT8 calibration table can be compatible, and the results are similar？

8

HI,
I use the TensorRT8.0 version on Xavier and directly set each layer dynamic range, but result still far from 2080ti’s int result,how can I solve it?and would you tell me what issues have been solved?

9

Hi,

The calibration results are not guaranteed to be transferrable between releases.
However, the most common cause of this phenomenon is different layer precisions decisions between the two engines.
We suggest you to please check this first.

Thank you.

10

thanks, according to my understanding, with the same model parameters and the same scaling factor, the output of the model should be the same. What are the reasons for this error? What tools are available to see the accuracy error between layers？

11

Hi,

Even if we have the same scaling factor, it’s not guaranteed that during runtime layers precision would be the same. Please verify layer precision in the logs.

Thank you.