Question Regarding INT8 Accuracy Degradation on Turing GPUs with TensorRT 10.11

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
1

Description

We observed a significant accuracy degradation when running INT8 engines on Turing GPUs (SM 7.5), while FP16 works properly on the same hardware.
This issue was discovered during our migration from TensorRT 8.6 to TensorRT 10.11.

Below are the test results:

External Image

Model Name Ada Ampere Blackwell Turing Precision Number of class Date
StandardModel-Core-10.0g2-Lite-3.4 0.4141 0.4191 0.4222 0.3002 int8 28 2026/05/08
PowerModel-Core-10.0g2-Pro-3.4 0.7607 0.7642 0.7750 0.6506 int8 28 2026/05/04
PowerModel-Core-10.0g2-Pro-3.4 0.8059 0.8028 0.8029 0.8058 fp16 28 2026/05/08
EM-SOPPE-Core-10.0g2-Pro-1.6.1 0.8556 0.8556 0.8556 0.8556 int8 2 2026/05/05
PS-SOFirearms-Core-10.0g2-Pro-3.2 0.9071 0.9071 0.9072 0.9072 int8 2 2026/05/05

  • FP16 inference works properly on Turing.

  • Only INT8 inference on Turing shows significant accuracy degradation in certain models, especially when the number of classes is higher.

  • Ampere / Ada / Blackwell do not show this issue.

From the results, the issue appears to affect only specific INT8 models on Turing GPUs.

We also noticed the following statement in the TensorRT 10.11 release notes:

“SM 7.5 and earlier devices may not have INT8 implementations for all layers with Q/DQ nodes. In this case, you will encounter a could not find any implementation error while building your engine. To resolve this, remove the Q/DQ nodes, which quantize the failing layers.”

Could someone help to clarify:

  1. Is this a known limitation/issue on Turing (SM 7.5) with TensorRT 10.11 INT8 + Q/DQ models?

  2. Any recommendation? or solution?

Any guidance or debugging recommendations would be greatly appreciated. A clear and concise description of the bug or issue.

Environment

TensorRT Version: 10.11
GPU Type: Turing
Nvidia Driver Version: 580.159.03
CUDA Version: 12.8
CUDNN Version: 9.10
Operating System + Version: Ubuntu 24.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): container with tag nvcr.io/nvidia/tritonserver:25.06-py3

2

*** Please note that this reply is generated by LLM automatically ***

The issue you’re experiencing with significant accuracy degradation when running INT8 engines on Turing GPUs (SM 75) is a known limitation in TensorRT 10.1.1.

According to the release notes, SM 75 and earlier devices may not have INT8 implementations for all layers with Q/DQ nodes. This can lead to a “could not find any implementation” error while building the engine.

The recommended solution is to remove the Q/DQ nodes that quantize the failing layers. This is because the Q/DQ nodes are used to control the compute and data precision of the network, and if the INT8 implementation is not available for a particular layer, it can cause accuracy degradation.

To resolve this issue, you can try the following:

  1. Remove the Q/DQ nodes that quantize the failing layers. This will allow the engine to use a different implementation for those layers.
  2. Use a different GPU architecture that supports INT8 implementations for all layers, such as Ampere or Ada.
  3. Use a different precision mode, such as FP16, which is supported on Turing GPUs.

It’s also worth noting that the issue appears to affect only specific INT8 models on Turing GPUs, and the number of classes in the model seems to be a factor in the accuracy degradation.

In terms of debugging recommendations, you can try the following:

  1. Check the engine logs for any errors or warnings related to the Q/DQ nodes or INT8 implementations.
  2. Use the TensorRT debugger to step through the engine execution and identify the specific layers that are causing the accuracy degradation.
  3. Try running the engine with different precision modes or GPU architectures to see if the issue is specific to INT8 on Turing GPUs.

Here is a comparison of the different GPU architectures and their support for INT8 implementations:

GPU Architecture INT8 Support
Turing (SM 75) Limited support for INT8 implementations, may not have implementations for all layers with Q/DQ nodes
Ampere Full support for INT8 implementations
Ada Full support for INT8 implementations

In summary, the issue you’re experiencing is a known limitation in TensorRT 10.1.1, and the recommended solution is to remove the Q/DQ nodes that quantize the failing layers or use a different GPU architecture that supports INT8 implementations for all layers.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

3

Our current model does not utilize the Q/DQ insertion approach. However, we have recently identified a feasible solution and can proceed accordingly. Thanks.