Building int8 model: combining direct dynamic range setting with calibration

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi,

I’m building a TensorRT engine for int8 precision, and have to use implicit quantization because I’m targeting the DLA. Is it possible to combine direct setting of dynamic range (via tensor.set_dynamic_range()) with calibration? There are some tensors that I know the dynamic range for, and can set a tighter range than I’d get through calibration, but for the rest of the tensors I want the dynamic range to come from the calibration data. In particular, if I directly set the dynamic range of some tensors using tensor.set_dynamic_range(), will that value be wiped out if I subsequently run calibration?

Thanks!

2

To answer my own question, it looks like you CAN combine direct precision setting and calibration. If you set the dynamic range manually, it will NOT be over-written by calibration. Seeing output like:

11/19/2024-12:41:44] [TRT] [V] User overriding scale with scale and zero-point Quantization(scale: {0.0472441,}, zero-point: {0,})
[11/19/2024-12:41:44] [TRT] [V] User overriding scale with scale and zero-point Quantization(scale: {0.0472441,}, zero-point: {0,})
[11/19/2024-12:41:44] [TRT] [V] INT8 Inference Tensor scales and zero-points: /network/backbone/maxpool/MaxPool_output_0 scale and zero-point Quantization(scale: {0.0472556,}, zero-point: {0,})
[11/19/2024-12:41:44] [TRT] [V] User overriding scale with scale and zero-point Quantization(scale: {0.0472441,}, zero-point: {0,})
[11/19/2024-12:41:44] [TRT] [V] User overriding scale with scale and zero-point Quantization(scale: {0.0472441,}, zero-point: {0,})

So it seems to be respecting the user precision override.