Worse performance after quantization on TensorRT

AI & Data Science Deep Learning (Training & Inference) TensorRT
, , ,
1

Description

Context: I am deploying the DEIMv2 model on a Jetson Orin NX (JetPack 6.2) and attempting to optimize performance using INT8 quantization via NVIDIA ModelOpt.

The Issue: While my FP16/BF16 baseline achieves 580 QPS, the quantized INT8 version drops to 380 QPS. I expected a significant speedup, but I am seeing a performance regression instead.

Current Workflow:

  1. PTQ: Applied Post-Training Quantization using modelopt.torch.quantization.

  2. Export: Exported to ONNX using OPSET 17 via the ModelOpt exporter.

  3. Sanitization: Used polygraphy surgeon to clean the graph.

  4. Engine Build: Compiled using trtexec with the --stronglyTyped flag enabled.

Questions:

  • Are there specific configurations in ModelOpt for DEIMv2 (Transformer-based) to ensure QDQ nodes are placed only in TensorRT-fusible locations?

  • Does the DEIMv2 ONNX graph require manual modification (e.g., folding specific constants or merging redundant reshapes) to avoid “reformat” nodes during engine generation?

  • Are there detailed guides for end-to-end quantization for Orin NX that cover the --stronglyTyped workflow specifically for Vision Transformers/DEIM?

Environment

TensorRT Version: 10.7
GPU Type: Jetson Orin NX
Nvidia Driver Version: 540.4.0
CUDA Version: 12.6
CUDNN Version:
Operating System + Version: L4T 34.4
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 2.7
Baremetal or Container (if container which image + tag): Container

Relevant Files

This is the github repo of the model: Intellindust-AI-Lab/DEIMv2: [DEIMv2] Real Time Object Detection Meets DINOv3

Steps To Reproduce

  • Quantize the model with ModelOpt mqt.quantize(model, cfg, forward_loop)

  • Export to ONNX using the exporter here Model-Optimizer/examples/onnx_ptq/download_example_onnx.py at main · NVIDIA/Model-Optimizer

  • [Optional] Saniteze ONNX with polygraphy surgeon sanitize in.onnx -o out.onnx --cleanup --toposort --fold-constant --no-onnxruntime-shape-inference

  • Generate engine with trtexec --onnx=${ONNX_FILE} \

        --saveEngine=${ENGINE_FILE} \\

    --exportLayerInfo=layer.json \\

    --profilingVerbosity=detailed \\

    --exportProfile=prof.json \\

    --stronglyTyped
3

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, I will attempt to address the question and provide a comprehensive answer.

Performance Regression with INT8 Quantization

The user is experiencing a performance regression when deploying a DEIMv2 Transformer-based model on a JetPack 62 with INT8 quantization using NVIDIA TensorRT. The FP16/BF16 baseline achieves a certain performance, but the quantized INT8 version drops to a lower performance.

Possible Causes and Recommendations

  1. Q/DQ Node Placement: The user is using ModelOpt to optimize the model, but it’s possible that the Q/DQ nodes are not placed in TensorRT-fusible locations. To ensure this, the user can try to manually modify the ONNX graph to place the Q/DQ nodes in fusible locations. The user can refer to the NVIDIA TensorRT documentation for guidelines on placing Q/DQ layers in the network.
  2. Manual Modification of ONNX Graph: The user may need to manually modify the ONNX graph to avoid reformat nodes during engine generation. This can be done using tools like ONNX-GraphSurgeon. The user can also try folding specific constants or merging redundant reshapes to simplify the graph.
  3. End-to-End Quantization Workflow: The user can refer to the NVIDIA TensorRT documentation for detailed guides on end-to-end quantization for Vision Transformers. The documentation provides a workflow specifically for Orin NX, which includes steps for quantizing the model, exporting to ONNX, and generating the engine.
  4. ModelOpt Configuration: The user can try to configure ModelOpt to optimize the model for INT8 quantization. This may involve setting specific flags or options to ensure that the Q/DQ nodes are placed in fusible locations.
  5. TensorRT Version and Compatibility: The user should ensure that they are using a compatible version of TensorRT with their JetPack 62 and Orin NX environment. The user can refer to the NVIDIA TensorRT documentation for compatibility information.

Comparison with Other Frameworks

The user is using PyTorch to train the model, but they are exporting the model to ONNX and using TensorRT for inference. This is a common workflow, but the user may want to consider using other frameworks like TensorFlow or PyTorch’s built-in quantization tools. However, since the user is already using TensorRT, it’s likely that they want to stick with the NVIDIA ecosystem.

Additional Recommendations

  1. Profile the Model: The user can try to profile the model using tools like NVIDIA Nsight Systems to identify performance bottlenecks.
  2. Optimize the Model: The user can try to optimize the model using techniques like pruning, distillation, or knowledge distillation to reduce the computational complexity of the model.
  3. Use FP16 or BF16: If the user is experiencing performance regressions with INT8 quantization, they may want to consider using FP16 or BF16 instead. These formats may provide better performance and accuracy trade-offs.

In summary, the user should try to manually modify the ONNX graph to place the Q/DQ nodes in fusible locations, configure ModelOpt to optimize the model for INT8 quantization, and refer to the NVIDIA TensorRT documentation for detailed guides on end-to-end quantization for Vision Transformers. Additionally, the user can try to profile the model, optimize the model, and use FP16 or BF16 instead of INT8 quantization.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***