With INT8 the accuracy of the network_api_pytorch_mnist model drops to less than 10 %

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
1

Hi all, this is w.r.t to the observation I have that the pytorch mnist model’s performance drops to less than 10% when INT8 calibration is introduced.

There is an existing sample.py in the directory /usr/src/tensorrt/samples/python/network_api_pytorch_mnist/
It has got support for FP32 and FP32 data type inference but not for INT8
I have modified the code sample.py to have INT8 calibration referring to the sample.py code present in the directory /usr/src/tensorrt/samples/python/int8_caffe_mnist/

Though it worked, but the accuracy got reduced to less than 10%
Out of 10000 inferences it could successfully predict

Predicted 916 / 10000 correctly
It means the accuracy is only 9.16 %
But the with FP32 and FP16 it gave more than 97% accuracy
Let me know what could be the reason and how to fix this issue.

Thanks and Regards

Nagaraj Trivedi

4

Dear @trivedi.nagaraj,
I think some issue with the INT8 calibration cache. Did you generate a new INT8 calibration cache? Also, may i know the Jetpack version? Just to confirm if you are using Jetson Orin Nano board?

5

Hi SivaRamaKrishna, yes a new INT8 calibration cache is generated. The Jetpack version begin used is 4.6.1 and the board is Jetson Nano.

Thanks and Regards
Nagaraj Trivedi

6

Hi SivaRamaKrishna, can you please update me on this query. I need this test to be conducted with INT8 calibration to compare the result with FP32 and FP16.

An early solution to this problem will definitely help me.

Thanks and Regards

Nagaraj Trivedi

7

Dear @trivedi.nagaraj,
Could you share details on how this new cache is generated? Did you use any dataset of images to generate cache?

8

present in the file /usr/src/tensorrt/samples/python/int8_caffe_mnist/sample.py

The dataset used to generate the cache is from the location
“/usr/src/tensorrt/data/mnist/t10k-images-idx3-ubyte”
Below three lines are added to generate calibration cache.
calibration_cache = “mnist_calibration1.cache”
test_set = “/usr/src/tensorrt/data/mnist/t10k-images-idx3-ubyte”
calib = MNISTEntropyCalibrator(test_set, cache_file=calibration_cache)

I am also attaching source files so that you can have a look.
The sample.py from the directory /usr/src/tensorrt/samples/python/network_api_pytorch_mnist/
got modified referring the sample.py file from the directory /usr/src/tensorrt/samples/python/int8_caffe_mnist/
I am attaching the files below files in the zip format.

  1. sample.py modified from network_api_pytorch_mnist/ directory
  2. The calibration cache file generated is mnist_calibration1.cache
  3. calibrator.py
  4. saved_mnist_model (which is a saved trained model file from model.py)
  5. model.py

The sample.py runs in three modes FP32, FP16 and INT8. It is controlled by the flag INFER_WEIGHT_SIZE. If it is set to “INT8” then it uses the INT8 calibration. Please refer to this flag in the sample.py

I am pasting below the inference results which I got with FP16 and INT8.

Inference with INT8 where it detects
Predicted 970 / 10000 correctly
It means only 970 images out of 10000 images are predicted correctly and the accuracy is less than 10%

But the inference with FP16 is
Predicted 9759 / 10000 correctly
It means the accuracy is 97.59%

Also I have a doubt with regards to weights I am assigning in the method
populate_network()
Please verify in the sample.py file whether I am assigning the weights to all the layers properly particularly when the inference mode is not FP16.

Thanks and Regards

Nagaraj Trivedi
saved_mnist_model.zip (12.7 MB)

9

Dear @trivedi.nagaraj,
Thank you for sharing the files. I will check and get back to you on this.

10

Dear @trivedi.nagaraj,
I don’t see MNIST INT8 python sample in Jetson Orin nano with Jetpack 5.1.2. Could you confirm the Jetpack version and platform?

11

Hi SivaRamaKrishnan, sorry I have wrongly mentioned the board info.

=== Device Information ===
[11/21/2023-05:10:08] [I] Selected Device: Xavier
[11/21/2023-05:10:08] [I] Compute Capability: 7.2

It is the Jetson Xavior 7.2 and the jetpack version is
Package: nvidia-jetpack
Version: 4.6-b199

Thanks and Regards

Nagaraj Trivedi

12

Hi ShivaRamaKrishnan, can you please update me on this. It is very much required for my academic study submission.

Thanks and Regards