INT8 Calibration in Python with TensorRT 8.6

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello,

I’m trying to quantize in INT8 YOLOX_Darknet from ONNX, using TensorRT 8.6 in Python.

Running it in TF32 or FP16 is totally fine.

I found various calibrators but they are all outdated and using apparently deprecated code, like :

-how to use tensorrt int8 to do network calibration | C++ Python. Computer Vision Deep Learning | KeZunLin's Blog (from 2018, error is “[ERROR] Exception caught in get_batch(): TypeError: PythonEntropyCalibrator.get_batch() missing 1 required positional argument: ‘names’” while building serialized network) → I searched and found this thread int8 calibration,meet error get_batch() takes 2 positional arguments but 3 were given, but the “newer” versions (seemingly from TRT 5) doesn’t work for me.

-tensorrt-utils/int8/calibration/SimpleCalibrator.py at master · rmccorm4/tensorrt-utils · GitHub (from 2020, but it uses “config.get_calibration_profile()” which uses directly the “config” while in my understanding the calibrator was to be added to the config and not the opposite)

-How to deploy an ONNX model with int8 calibration? · Issue #557 · NVIDIA/TensorRT · GitHub This thread gives dead link but my question is very close from this one, and at the time answers were available.

-TensorRT/samples/python/int8_caffe_mnist/calibrator.py at 498dcb009fe4c2dedbe9c61044d3de4f3c04a41b · NVIDIA/TensorRT · GitHub is the most official python sample I found, I tried to tweak it’s not super clear what is missing or not working.

-Finally, TensorRT/samples/sampleINT8 at master · NVIDIA/TensorRT · GitHub is the official sample but it’s in C++.

So my question is, what should be in a calibrator, and do NVIDIA has an up to date example ? the documentation (Developer Guide :: NVIDIA Deep Learning TensorRT Documentation) is pretty small so I’d like to have more informations about how to create a calibrator in python.

Environment

TensorRT Version: 8.6
GPU Type: RTX3060
Nvidia Driver Version: latest as of 23/06/2023
CUDA Version: 11.8
Operating System + Version: Win11
Python Version (if applicable): 3.10

Relevant Files

Official YOLOX_darknet ONNX from YOLOX repo : YOLOX/demo/ONNXRuntime at main · Megvii-BaseDetection/YOLOX · GitHub

Steps To Reproduce

Create a script to quantize the model from ONNX to TRT using INT8 precision.

2

Hi, Please refer to the below links to perform inference in INT8

Thanks!

5

Hello,

Thank you for your answer, but I already mentionned this github in my original post, the sample is only in C++.
I guess this means that there is no up-to-date sample in Python ?

And is it even possible to do it in Python with TensorRT 8.6, as many functions are now deprecated ?

Thanks

6

Hi,

Please to the following samples (build_engine.py) which may help you.

Thank you.

7

Thank you ! Yeah the beginning of the script in build_python.py is exactly what I was looking for, an up-to-date sample of calibator to see what is expected in python.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.