TensorRT sampleINT8API Demo low accuracy

Description

I run the TensorRT Demo sampleINT8API, and i get result a little different from: https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/
I get:

[I] sampleINT8API result: Detected:
[I] [1] airliner
[I] [2] warplane
[I] [3] space shuttle
[I] [4] projectile
[I] [5] wing

and the demo shows:

[I] sampleINT8API result: Detected:
[I] [1] space shuttle
[I] [2] airliner
[I] [3] warplane
[I] [4] projectile
[I] [5] wing

And i resize all validation set of image_net(50000 images) with the dimensions of 224x224x3 and convert them to PPM extension. But after i run all of the validation set , the top-1/top5 accuracy are only 53.61% and 78.42%, anything wrong?

Environment

TensorRT Version: 7.0:
GPU Type: T4:
Nvidia Driver Version: 410.79:
CUDA Version: 10.0:
**CUDNN Version: 7.6.4:
Operating System + Version: CentOS 7.6.1810:
Python Version (if applicable): 2.7:
TensorFlow Version (if applicable): 1.15:

Relevant Files

Demo:

Hi,

Regarding different result:
We have a slight change on the model and maybe caused this. It seems documentation is not updated with new results.

Regarding calibration accuracy:
Could you please first try to calibrate on 100 images only?
~/data/samples/int8_api/resnet50_per_tensor_dynamic_range.txt

Thanks

Hi, thanks for your response, when i run the model without INT8
quantify, i get top1/top5: 55.4%/80%(resize image_net images to 224x224x3 and convert them to PPM extension).

So i tried, i first resize every image’s shorter edge to 244, then i crop center 244x244x3 and convert them to PPM extension, i get 61.63%/84.8%.
I also try like https://github.com/onnx/models/blob/master/vision/classification/imagenet_inference.ipynb, they resize images to size 256x256, take center crop of size 224x224, where gives the resnet model the demo uses, however i get poor result.

How can i get the right result?

Hi,

The data is not calibrated for whole data set in this sample, usually it’s only 100 inputs.
~/data/samples/int8_api/resnet50_per_tensor_dynamic_range.txt
Did you try to calibrate the model on first 100 images?

Thanks