Description A clear and concise description of the bug or issue. Environment TensorRT Version: TensorRT 8.0.1 GPU Type: RTX 3070 Nvidia Driver Version: 470.63.01 CUDA Version: 11.3 CUDNN Version: 8.2.2.26 Operating System + Version: Ubuntu 20.04 Python Version (if applicable): 3.6 TensorFl…

TRT Engin in INT8 is much slower than FP16

AI & Data Science Deep Learning (Training & Inference) TensorRT

spolisetty November 3, 2021, 7:56am 5

Hi,

It’s possible if many layers end up falling back to fp32. You’d probably want to enable both int8 and fp16 and check.
Could you please share trtexec --verbise logs for both FP16 and INT8 mode commands.

Thank you.

Topic		Replies	Views
QAT int8 TRT engine slower than fp16 TensorRT tensorrt , pytorch , python , onnx	3	2252	January 6, 2022
TRT Uses INT 32 VS INT 16 TensorRT	3	1007	October 12, 2021
INT8 (8-bit inference, post-training quantization) on Windows 10 is much slower than Ubuntu 20.04 TensorRT	5	740	September 23, 2022
TensorRT int8 slower than FP16 due to reformat layer TensorRT tensorrt , cudnn	0	86	October 11, 2024
Int8 performance is less than fp16 TensorRT tensorrt	3	860	September 2, 2022
TensorRT INT8 inference accuracy TensorRT	2	500	May 9, 2022
ONNX Model Int64 Weights TensorRT	12	13356	February 17, 2024
Same inference speed for INT8 and FP16 TensorRT	10	5806	October 12, 2021
How to apply int8 quantization to Transformer on Xavier Jetson Xavier NX tensorrt	2	481	August 12, 2022
Int8 get the same result, but in FP16 the result is correct TensorRT	1	398	December 1, 2021

TRT Engin in INT8 is much slower than FP16

Related topics