INT8 quantization

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I recently tried the TF-TRT script for INT8 quantization. But, I did not get the calib_tables. The calib_table files are empty. I used automatic quantization of TF-TRT feature (using the calibrate function provide by the converter). I am unable to attach the frozen graph that Im trying. But I am wondering if there are any conditions to be met for calibration? (like a specific NVIDIA hardware or CUDA version) . I tried using the latest nvidia tensorflow container (19.10-py3).

2

Hi,
Can you provide the following information so we can better help?
Provide details on the platforms you are using:
o Linux distro and version
o GPU type
o Nvidia driver version
o CUDA version
o CUDNN version
o Python version [if using python]
o Tensorflow version
o TensorRT version
o If Jetson, OS, hw versions

Also, if possible please share the script that you are using for INT8 quantization.

Meanwhile, please look into below links for steps and sample examples:

https://github.com/tensorflow/tensorrt/tree/master/tftrt/examples/image-classification

Thanks

3

The GPU I use is Quadro GV100 with driver version : 430.50 and CUDA version 10.1.

All other info is already provided when I said that I use the container “tensorflow:19.10-py3”, I hope.

I’ll soon check if I can share the script and the input graph with you. In the meantime, it would be nice if you provide sample scripts for simple examples (something like hello world). Then I might read the above example you mentioned.

4

Hi,

Please refer to the below link for examples and user guide:

Thanks

5

I am aware of these things. I am asking for simple examples (e.g. one matmul). Once people set up those examples, then they can follow tougher ones like object detection or image classification. I know that it has become typical of vendors to put the ball in the customer’s court and ask them to do everything…I won’t be surprised if I get a response that “go ahead and test it for a simple matmul yourself”. But, it is the complex dependencies on hardware and software that’s causing the current situation. So, if you put a simple example (e.g. matmul) in the container and show steps how it works, then it’ll become clear to me how to extend it for my own examples.

6

Hi,
You can also refer to below test scripts for as additional samples:
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/compiler/tensorrt/test

Thanks