Alexnet using INT8


I have created alexnet with TensorRT using the api creation instead of caffe model and parsing . I trained weights,biases which FLOAT format for alexnet.

My query is ,Is there anyway TensorRT quantizes them(weights/biases) to half-precision/UINT8 and verify the Alexnet in half-precision/UINT8 which would be very much useful? If so what are the settings need to be made.

Thanks in advance


I have found that sample_INT8 does exist which shows the example for mnist.
Can you let me know if my understanding with sample int8 is correct

1)what is the use of BatchStream calibrationStream(CAL_BATCH_SIZE, NB_CAL_BATCHES);
2) I assume the Int8EntropyCalibrator calibrator(calibrationStream, FIRST_CAL_BATCH); does the actual calibration . But how the data need to be feed for calibrator. I assume that batches folder need to be created and then batch0,batch1… need to be created and data need to be copied in it. But it is a tedious task.
3)scoreModel(batchSize, firstScoreBatch, nbScoreBatches, &calibrator);
I assume this is the one which does the actual scoring and lets us know whether INT8 weights are useful for our usage or not for inference . Where the INT8 converted weights will be placed ?

Questions may appear too basic? If there is any documentation regarding the same it would much more helpful for the newbies…

Thanks in advance…

I would also like to see more documentation on these - I’m currently trying to suss out what actually happens within these same steps, and the docs on the Int8EntropyCalibrator are particularly lacking. I’m interested to know what exactly the calibrator does with the calibration stream.

I would also like to know more about Calibrator and also how the quantization method works in TensorRT . If you guys have figured it out something by now , please share…Thanks!

Sorry, this comment was moved to another place

Hi Subaarukun,

Were u able to run Alexnet/ Imagenet with INT8 with TENSORRT?
However, I am using only 2 classes Dogs vs Cats in caffe on GTX 1080 TI

Can you please share your results

I got below error:

TensorRT-2.1.2/bin> ./sample_int8 imagenet

INT8 run:4 batches of size 10 starting at 10
cudnnEngine.cpp (357) - Cuda Error in execute: 77