NVIDIA Developer Forums

Post quantization aware training is slower than fp16 and post quantization

AI & Data Science Deep Learning (Training & Inference) TensorRT

OnePieceOfDeepLearning September 27, 2021, 3:46am 7

int8 is supported in jetson nx. we got very great speed with PTQ.

QAT int8 TRT engine slower than fp16

Topic		Replies	Views	Activity
QAT int8 TRT engine slower than fp16 TensorRT tensorrt , pytorch , python , onnx	3	2310	January 6, 2022
TensorRT --fp16 pre and post Int8 quantization TensorRT cudnn	1	108	September 2, 2024
[Hugging Face transformer models + pytorch_quantization] PTQ quantization int8 is slower than fp16 TensorRT tensorrt , python , onnx , natural-language-processing-nlp	4	3046	January 6, 2022
How can we know we have convert the onnx to int8trt rather than Float32? TensorRT tensorrt	23	1917	June 14, 2021
How to verify if QAT TRT engine is indeed INT8 on Xavier Jetson AGX Xavier tensorrt	16	604	October 5, 2022
Achieving FP32 Accuracy for INT8 Inference Using Quantization Aware Training with NVIDIA TensorRT Technical Blog	1	848	December 3, 2023
Post-Training Quantization (PTQ) for semantic segmentation model running on Jetson Orin NX Jetson Orin NX tensorrt	24	344	March 26, 2025
TRT Engin in INT8 is much slower than FP16 TensorRT	4	1974	November 11, 2021
TensorRT the inference is slow for the QAT model comparing to the PTQ case Jetson AGX Xavier tensorrt , nvbugs	19	1625	January 16, 2023
Data inferencing to INT8U quantized model TensorRT tensorrt	2	418	October 12, 2021