ONNX Model INT8 Engine Build

AI & Data Science Deep Learning (Training & Inference) TensorRT
, , ,
1

Description

I’ve successfully build engines by using prototxt file with INT8 calibrations. Besides, when I use ONNX models with FP16 data, I can also build engines. However, I’m receiving an error with ONNX Model + INT8 calibration. This sounds like a simple error however, I can’t find the source of it. The message is attached below:

Note: I’ve attached the files below. I’m aware of calibrating values are not correct. However, I don’t focus on the accuracy of the model at this point but getting some runtimes with INT8.

Error Message

INFO:EngineBuilder:Using calibration cache file: tensorrt_scripts/calibrator_networks/pose_densenet121_body_calibration
[07/20/2022-17:09:52] [TRT] [E] 4: [standardEngineBuilder.cpp::initCalibrationParams::1398] Error Code 4: Internal Error (Calibration failure occurred with no scaling factors detected. This could be due to no int8 calibrator or insufficient custom scales for network layers. Please see int8 sample to setup calibration correctly.)
[07/20/2022-17:09:52] [TRT] [E] 2: [builder.cpp::buildSerializedNetwork::620] Error Code 2: Internal Error (Assertion engine != nullptr failed. )

Environment

Jetson AGX Orin with Jetpack 5.0.1. Specifically, TensorRT 8.4.0

Relevant Files

I’m using Prototxts files and ONNX models from jetson-inference repo.
engine_building.py.py (5.2 KB)
pose_densenet121_body_calibration (8.5 KB)
pose_densenet121_body.onnx (79.4 MB)

Steps to Reproduce

I’ve attached the code and the model just in case if you need to reproduce. Please dont forget to update the file paths.

2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

I believe I shared at the top what you requested. Was this message an auto-message or is there anything that is not clear in my post?

Thanks!

5

Hi,

Could you please give us more details on how you generated the calibration cache.

If you’re doing calibration cache generation wrongly, this may occur.
Please make sure calibration algorithm runs before the fusions occur (this way every tensor should have a scale attached to it). Calibrate an FP32/FP16 model using some data, this will generate a collection of scaling factors for each layer.

Please refer to the following sample.

https://github.com/NVIDIA/TensorRT/tree/main/samples/sampleINT8#calibration-file

Thank you.