INT8 calibration cache doesn't created

user102103 · February 25, 2022, 6:42am

Description

Hi, I’m trying to build my centerNet model with INT8 engine.

I have built INT8 engine, but the calibration cache was not created.
I have ran it several times using tensorrt.Builder.build_engine() or tensorrt.Builder.build_cuda_engine(), but calibration cache was still not created.

I think write_calibration_cache() method wasn’t called, but I have no idea why this happen.
The API seems to be a little different between the version I use and the latest version of sensorrt. Are there some example for tensorrt 7.2.x?
I’d appreciate a lot if someone could give me advice.

And I have another question: int8 mode and calibrator can be set in tensort.Builder. Also, int8 mode and calibrator can be set in tensorrt.IBbuilderConfig. The difference between that two option seems to be tensorrt.Builder.build_engine() and tensorrt.Builder.build_cuda_engine(). What is the difference between the two ways to build an engine?

Environment

TensorRT Version: 7.2.2.1
GPU Type: Nvidia Titan Xp
Nvidia Driver Version:
CUDA Version: 11.1
CUDNN Version: 8.0.5
Operating System + Version: CentOS 7.9.2009
Python Version (if applicable): 3.8.5
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.9.0
Baremetal or Container (if container which image + tag): container (nvcr.io/nvidia/tensorrt:20.12-py3)

NVES · February 25, 2022, 4:07pm

Hi, Please refer to the below links to perform inference in INT8

Thanks!

user102103 · March 2, 2022, 4:51am

Thanks. But I have reviewed the sample you shared.
I am using python3 with onnx, Is there a sample that fits that environment?

Or, is there a sample about TensorRT 7.2.x?

spolisetty · March 24, 2022, 6:24am

Hi,

We have only the following samples. You can take reference and try for the ONNX model.
https://github.com/NVIDIA/TensorRT/blob/main/samples/python/int8_caffe_mnist/sample.py#L44

github.com

NVIDIA/TensorRT/blob/main/samples/python/introductory_parser_samples/onnx_resnet50.py#L41


      
          
          
          class ModelData(object):
              MODEL_PATH = "ResNet50.onnx"
              INPUT_SHAPE = (3, 224, 224)
              # We can convert TensorRT data types to numpy types with trt.nptype()
              DTYPE = trt.float32
          
          
          # You can set the logger severity higher to suppress messages (or lower to display more messages).
          TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
          
          # The Onnx path is used for Onnx models.
          def build_engine_onnx(model_file):
              builder = trt.Builder(TRT_LOGGER)
              network = builder.create_network(common.EXPLICIT_BATCH)
              config = builder.create_builder_config()
              parser = trt.OnnxParser(network, TRT_LOGGER)
          
              config.max_workspace_size = common.GiB(1)
              # Load the Onnx model and parse it in order to populate the TensorRT network.

Thank you.

Topic		Replies	Views
ONNX Model INT8 Engine Build TensorRT tensorrt , jetson-inference , calibration , onnx	3	1907	July 26, 2022
Problem related saving calibration cache TensorRT	5	1007	October 12, 2021
TensorRT INT8 conversion from an ONNX model TensorRT tensorrt , calibration , onnx	4	5455	July 29, 2024
INT8 calibration file not generating, not building in INT8 mode TensorRT tensorrt , ubuntu , python , jetson-nano	15	2422	June 4, 2022
INT8 Calibration in Python with TensorRT 8.6 TensorRT tensorrt	5	3606	July 12, 2023
Tenssorrt INT8 precision engine build failed for the models having custom layer (BatchedNMSDynamic_TRT) TensorRT	11	1911	June 29, 2021
INT8 calibration cache for trtexec TensorRT tensorrt , cudnn , calibration , onnx	0	75	July 29, 2024
How to generate int8 calilb table for trtexec engine generation TensorRT tensorrt	7	4424	October 12, 2021
Python API - int8_calibrator not used when calling build_engine (but works when calling build_cuda_engine) TensorRT python	3	849	October 12, 2021
Int8 calibration TensorRT	1	2348	December 17, 2021

INT8 calibration cache doesn't created

Description

Environment

Related topics