TRT Engin in INT8 is much slower than FP16


A clear and concise description of the bug or issue.


TensorRT Version: TensorRT 8.0.1
GPU Type: RTX 3070
Nvidia Driver Version: 470.63.01
CUDA Version: 11.3
CUDNN Version:
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): 3.6
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.7
Baremetal or Container (if container which image + tag): Baremetal

Relevant Files


Steps To Reproduce

  1. Convert ONNX to TRT model in FP16 and INT8
./trtexec --onnx=model.onnx --minShapes=input0:1x1x1024x256 --optShapes=input0:1x1x1024x500 --maxShapes=input0:1x1x1024x650 --fp16 --workspace=5000 --verbose --saveEngine=model_fp16.bin
./trtexec --onnx=model.onnx --minShapes=input0:1x1x1024x256 --optShapes=input0:1x1x1024x500 --maxShapes=input0:1x1x1024x650 --int8 --calib=calibration.cache  --workspace=5000 --verbose --saveEngine=model_int8.bin
  1. Infer data with these models
    It shows that FP16 is much faster than INT8 model:
Time Used for model model_fp16.bin: 6.716s
Time Used for model model_int8.bin: 15.277s

Please help me on this.
Thank you.


Hi, Please refer to the below links to perform inference in INT8


Thank you. I checked the steps I used in converting ONNX model to TRT Int8 model, it is compliance with the steps you provided. But the model is much slower in INT8 than the one in FP16.


It’s possible if many layers end up falling back to fp32. You’d probably want to enable both int8 and fp16 and check.
Could you please share trtexec --verbise logs for both FP16 and INT8 mode commands.

Thank you.

Thank you. I checked the output with --verbose, found the fallback to FP32. So it explains the reason why int8 model would be slower than FP16.
Thank you!


1 Like