TensorRT sampleINT8API Demo low accuracy


I run the TensorRT Demo sampleINT8API, and i get result a little different from: https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/
I get:

[I] sampleINT8API result: Detected:
[I] [1] airliner
[I] [2] warplane
[I] [3] space shuttle
[I] [4] projectile
[I] [5] wing

and the demo shows:

[I] sampleINT8API result: Detected:
[I] [1] space shuttle
[I] [2] airliner
[I] [3] warplane
[I] [4] projectile
[I] [5] wing

And i resize all validation set of image_net(50000 images) with the dimensions of 224x224x3 and convert them to PPM extension. But after i run all of the validation set , the top-1/top5 accuracy are only 53.61% and 78.42%, anything wrong?


TensorRT Version: 7.0:
GPU Type: T4:
Nvidia Driver Version: 410.79:
CUDA Version: 10.0:
**CUDNN Version: 7.6.4:
Operating System + Version: CentOS 7.6.1810:
Python Version (if applicable): 2.7:
TensorFlow Version (if applicable): 1.15:

Relevant Files



Regarding different result:
We have a slight change on the model and maybe caused this. It seems documentation is not updated with new results.

Regarding calibration accuracy:
Could you please first try to calibrate on 100 images only?


Hi, thanks for your response, when i run the model without INT8
quantify, i get top1/top5: 55.4%/80%(resize image_net images to 224x224x3 and convert them to PPM extension).

So i tried, i first resize every image’s shorter edge to 244, then i crop center 244x244x3 and convert them to PPM extension, i get 61.63%/84.8%.
I also try like https://github.com/onnx/models/blob/master/vision/classification/imagenet_inference.ipynb, they resize images to size 256x256, take center crop of size 224x224, where gives the resnet model the demo uses, however i get poor result.

How can i get the right result?


The data is not calibrated for whole data set in this sample, usually it’s only 100 inputs.
Did you try to calibrate the model on first 100 images?