Hi there, I tried to benchmark int8 and fp16 for mobilenet0.25+ssd in jetson nx with jetpack 4.6. for post training, i use pytorch-quantization toolkit ( TensorRT/tools/pytorch-quantization at master · NVIDIA/TensorRT · GitHub ) and generate the calibrated onnx. But I found out the performance of i…

Post quantization aware training is slower than fp16 and post quantization

spolisetty September 27, 2021, 2:49am 6

Hi,

Looks like you’re using Jetson platform, May be INT8 is not supported on Jetson hardware, please check preceison support matrix here.

Please allow us some time to test it on V100. Meanwhile we recommend you to please try mixed precision and fp16.

Thank you.

Topic		Replies	Views
QAT int8 TRT engine slower than fp16 TensorRT tensorrt , pytorch , python , onnx	3	2162	January 6, 2022
Quantize model with pytorch-quantization Jetson Orin NX pytorch , generative_ai	4	699	April 3, 2024
Accelerating Quantized Networks with the NVIDIA QAT Toolkit for TensorFlow and NVIDIA TensorRT Technical Blog	0	384	June 16, 2022
Unable to build model engine for INT8 yolov8m quantized using tensorrt model optimizer TensorRT jetson , deepstream	5	149	September 24, 2024
INT8 calibration file not generating, not building in INT8 mode TensorRT tensorrt , ubuntu , python , jetson-nano	15	2345	June 4, 2022
How can we know we have convert the onnx to int8trt rather than Float32? TensorRT tensorrt	23	1856	June 14, 2021
Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT Technical Blog	1	814	December 3, 2023
How to verify if QAT TRT engine is indeed INT8 on Xavier Jetson AGX Xavier tensorrt	16	584	October 5, 2022
Sparsity in INT8: Training Workflow and Best Practices for NVIDIA TensorRT Acceleration Technical Blog	0	365	May 26, 2023
Same inference speed for INT8 and FP16 TensorRT	10	5688	October 12, 2021