TensorRT Model Optimizer INT8 quantization causes 2.7x performance regression on Jetson Orin Nano 4GB (ViT-S + DPT architecture)

suryatej · January 17, 2026, 1:32am

TensorRT Model Optimizer INT8 quantization causes 2.7x performance regression on Jetson Orin Nano 4GB (ViT-S + DPT architecture)

Environment

Hardware: Jetson Orin Nano 4GB Developer Kit
L4T Version: R36.4.3 (Jan 8, 2025)
JetPack Version: 6.2
TensorRT Version: 10.3.0
Model Optimizer Versions Tested: 0.31, 0.35, 0.40

Conversion Pipeline

Model exported to ONNX from PyTorch on desktop (x86)
INT8 quantization applied using TensorRT Model Optimizer on desktop
ONNX transferred to Jetson Orin Nano
TensorRT engine built on-device using trtexec

Model Architecture

ViT-S + DPT encoder for dense feature extraction.

Input: 1x1x364x490 (grayscale image, batch of 2 processed together)
Output: Multi-scale feature maps (1x2x32x364x490 for full encoder, or per-layer features for ViT-S only)

Description

When quantizing the ViT-S + DPT encoder to INT8 using NVIDIA TensorRT Model Optimizer, the resulting engine exhibits:

~2.7x performance regression compared to FP16 baseline
Compilation failure on ModelOpt 0.40 (latest version)
Even with trtexec’s native `–fp16 --int8` flags (bypassing ModelOpt), INT8 shows minimal improvement over FP16 (~5% vs expected 20-50%)Results Summary

Full Encoder (ViT-S + DPT)

Precision Mode	ModelOpt 0.31	ModelOpt 0.35	ModelOpt 0.40
FP16 only	41.8 ms	41.9 ms	42.7 ms
FP16+INT8 (trtexec flags)	39.7 ms	39.7 ms	40.4 ms
ModelOpt INT8 (`--stronglyTyped`)	114.5 ms ❌	114.1 ms ❌	FAILS ❌

ViT-S Only (ModelOpt 0.40)

Precision Mode	Latency	Throughput
FP16 only	30.75 ms	32.1 qps
FP16+INT8 (trtexec flags)	30.80 ms	32.5 qps
ModelOpt INT8 (`--stronglyTyped`)	30.34 ms	32.9 qps

Even the isolated ViT-S shows only ~1.3% improvement with INT8 quantization.

ModelOpt 0.40 Compilation Failure

trtexec \
  --onnx=model_backbone_modelopt40_int8.onnx \
  --stronglyTyped \
  --noDataTransfers \
  --useCudaGraph \
  --useSpinWait

Error:

[01/16/2026-16:22:07] [E] Error[2]: [weightsPtr.h::values::182] Error Code 2: Internal Error (Assertion type() == expectedDataType<T>() failed. )
[01/16/2026-16:22:07] [E] Engine could not be created from network
[01/16/2026-16:22:07] [E] Building engine failed
[01/16/2026-16:22:07] [E] Failed to create engine from model or file.
[01/16/2026-16:22:07] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v100300]

Steps to Reproduce

FP16 Baseline

trtexec \
  --onnx=model_backbone.onnx \
  --fp16 \
  --noDataTransfers \
  --useCudaGraph \
  --useSpinWait

ModelOpt INT8

python3 \
-m modelopt.onnx.quantization \
--onnx_path model_backbone.onnx \
--quantize_mode int8 \
--output_path model_backbone_int8.onnx


trtexec \
  --onnx=model_backbone_int8.onnx \
  --stronglyTyped \
  --noDataTransfers \
  --useCudaGraph \
  --useSpinWait

Attachments

Drive folder with all the onnx

AastaLLL · January 19, 2026, 5:25am

Hi,

We quantize the model and run it with trtexec on Orin Nano and can get 41 qps.
Could you check if this can meet your requirement?

$  python3 -m modelopt.onnx.quantization --onnx_path vits.onnx --quantize_mode int8 --output_path int8_model.onnx
$ /usr/src/tensorrt/bin/trtexec --onnx=int8_model.onnx --int8 --fp16
...
[01/19/2026-05:19:40] [I] === Performance summary ===
[01/19/2026-05:19:40] [I] Throughput: 41.3892 qps
[01/19/2026-05:19:40] [I] Latency: min = 24.355 ms, max = 24.4976 ms, mean = 24.4676 ms, median = 24.4681 ms, percentile(90%) = 24.4811 ms, percentile(95%) = 24.4847 ms, percentile(99%) = 24.4958 ms
[01/19/2026-05:19:40] [I] Enqueue Time: min = 0.801758 ms, max = 1.14743 ms, mean = 0.850748 ms, median = 0.835938 ms, percentile(90%) = 0.894531 ms, percentile(95%) = 0.93103 ms, percentile(99%) = 1.09009 ms
[01/19/2026-05:19:40] [I] H2D Latency: min = 0.0443115 ms, max = 0.0612793 ms, mean = 0.0482149 ms, median = 0.0476074 ms, percentile(90%) = 0.0505981 ms, percentile(95%) = 0.0562744 ms, percentile(99%) = 0.06073 ms
[01/19/2026-05:19:40] [I] GPU Compute Time: min = 23.949 ms, max = 23.9971 ms, mean = 23.969 ms, median = 23.9688 ms, percentile(90%) = 23.9817 ms, percentile(95%) = 23.9851 ms, percentile(99%) = 23.9954 ms
[01/19/2026-05:19:40] [I] D2H Latency: min = 0.343262 ms, max = 0.456421 ms, mean = 0.450377 ms, median = 0.451202 ms, percentile(90%) = 0.453613 ms, percentile(95%) = 0.454376 ms, percentile(99%) = 0.456299 ms
[01/19/2026-05:19:40] [I] Total Host Walltime: 3.06844 s
[01/19/2026-05:19:40] [I] Total GPU Compute Time: 3.04406 s

Thanks.

suryatej · January 19, 2026, 7:17pm

Hey @AastaLLL, thanks for replying? I have a few questions:

What tensorrt and model opt version are you using.
Can you help compile the other model i uploaded (full encoder) as that is what we ultimately care about.

Also just want to confirm that our experiments are on Jetson Orin Nano 4GB.

AastaLLL · January 21, 2026, 6:34am

Hi

Our Orin Nano 8GB device is set up with JetPack 6.2.1, so the TensorRT version is 10.3.
We pull the latest Model Optimizer directly.

What is the difference between the models?
We test the vits.onnx model only, since other files seem to have been quantized with Model Optimizer already.

Thanks.

suryatej · January 22, 2026, 3:36am

Hey @AastaLLL,

As i mentioned in the ticket, we are trying to run these models on Orin Nano 4GB.
The vits.onnx is a smaller version of the actual model we want to convert. We were able to run this with int8 precision but we notice no speedup in doing so. The actual model doesn’t even convert with the latest modelopt.
The quantized modelopt models are provided so you can reproduce the error on your end and debug the compilation failure further.

AastaLLL · January 22, 2026, 7:06am

Hi,

Could you also attach the original actual model?

Thanks.

suryatej · January 23, 2026, 2:37am

Okay, the models are present on the drive. Each modelopt version requires a slightly different min onnx so the models are different for 31,35 and 40.

AastaLLL · January 23, 2026, 7:08am

Hi,

Thanks for explaining this.
Sorry that we originally thought the backbone model was the converted output from Model Optimizer.
So only test the ViT-S one.

We will check this further and get back to you soon.
Thanks.

AastaLLL · February 13, 2026, 3:28am

Hi,

Thanks for your patience.
Here are our recent updates for you.

1.

We quantize the model with Mode Optimizer 0.41.0.
The int8 model cannot work with the default TensorRT 10.3, with the following error:

[01/27/2026-16:04:12] [E] Error[2]: [weightsPtr.h::values::182] Error Code 2: Internal Error (Assertion type() == expectedDataType<T>() failed. )

But it can work normally with our internal TensorRT release.
So please wait for our future TensorRT release for the fix.

2.

With the above int8 model, we don’t see the 2.7x performance regression.
Instead, the performance between int8 and fp16 are quite similiar:

FP16

[02/12/2026-01:56:53] [I] Throughput: 28.1819 qps
[02/12/2026-01:56:53] [I] Latency: min = 36.4265 ms, max = 36.7357 ms, mean = 36.6772 ms, median = 36.6803 ms, percentile(90%) = 36.7141 ms, percentile(95%) = 36.715 ms, percentile(99%) = 36.7357 ms
[02/12/2026-01:56:53] [I] Enqueue Time: min = 1.01276 ms, max = 1.71167 ms, mean = 1.08244 ms, median = 1.05368 ms, percentile(90%) = 1.16745 ms, percentile(95%) = 1.20776 ms, percentile(99%) = 1.71167 ms
[02/12/2026-01:56:53] [I] H2D Latency: min = 0.0428467 ms, max = 0.0772705 ms, mean = 0.0500337 ms, median = 0.0484543 ms, percentile(90%) = 0.0579834 ms, percentile(95%) = 0.0625 ms, percentile(99%) = 0.0772705 ms
[02/12/2026-01:56:53] [I] GPU Compute Time: min = 35.0419 ms, max = 35.1392 ms, mean = 35.0837 ms, median = 35.0845 ms, percentile(90%) = 35.1147 ms, percentile(95%) = 35.1243 ms, percentile(99%) = 35.1392 ms
[02/12/2026-01:56:53] [I] D2H Latency: min = 1.30298 ms, max = 1.55908 ms, mean = 1.54347 ms, median = 1.54611 ms, percentile(90%) = 1.55139 ms, percentile(95%) = 1.552 ms, percentile(99%) = 1.55908 ms
[02/12/2026-01:56:53] [I] Total Host Walltime: 3.12257 s
[02/12/2026-01:56:53] [I] Total GPU Compute Time: 3.08737 s

INT8:

[02/12/2026-02:40:54] [I] Throughput: 29.0629 qps
[02/12/2026-02:40:54] [I] Latency: min = 35.3667 ms, max = 35.4888 ms, mean = 35.4452 ms, median = 35.4452 ms, percentile(90%) = 35.4648 ms, percentile(95%) = 35.4738 ms, percentile(99%) = 35.4888 ms
[02/12/2026-02:40:54] [I] Enqueue Time: min = 1.48633 ms, max = 1.89337 ms, mean = 1.54987 ms, median = 1.53223 ms, percentile(90%) = 1.59692 ms, percentile(95%) = 1.63062 ms, percentile(99%) = 1.89337 ms
[02/12/2026-02:40:54] [I] H2D Latency: min = 0.046936 ms, max = 0.065918 ms, mean = 0.0508698 ms, median = 0.0501709 ms, percentile(90%) = 0.0534668 ms, percentile(95%) = 0.0595703 ms, percentile(99%) = 0.065918 ms
[02/12/2026-02:40:54] [I] GPU Compute Time: min = 33.9907 ms, max = 34.0718 ms, mean = 34.0271 ms, median = 34.0264 ms, percentile(90%) = 34.0444 ms, percentile(95%) = 34.0532 ms, percentile(99%) = 34.0718 ms
[02/12/2026-02:40:54] [I] D2H Latency: min = 1.30103 ms, max = 1.37488 ms, mean = 1.36725 ms, median = 1.36804 ms, percentile(90%) = 1.37134 ms, percentile(95%) = 1.37207 ms, percentile(99%) = 1.37488 ms
[02/12/2026-02:40:54] [I] Total Host Walltime: 3.09673 s
[02/12/2026-02:40:54] [I] Total GPU Compute Time: 3.06244 s

3.

We also analyze whether it is possible to get further improvement with int8.
But for this particular case, the gain of the underlying kernel from INT8 is limited.
So we don’t think it is possible to improve further.

Thanks.

Topic		Replies	Views
TensorRT Quantization for Jetson Inference Jetson Orin Nano tensorrt , jetson-inference , jetson , tensorrt-model-optimizer	3	624	October 17, 2025
Post-Training Quantization (PTQ) for semantic segmentation model running on Jetson Orin NX Jetson Orin NX tensorrt	23	906	March 12, 2025
Unable to build model engine for INT8 yolov8m quantized using tensorrt model optimizer TensorRT jetson , deepstream	5	753	September 24, 2024
Worse performance after quantization on TensorRT TensorRT cudnn , onnx , tensorrt-model-optimizer , jetson-orin	1	339	December 23, 2025
TensorRT quantization bug on Jetpack 6.0 Jetson AGX Orin tensorrt , pytorch	5	838	January 22, 2024
Benchmarck int8 similar to fp32 on yolov8 from ultralytics Jetson Orin Nano tensorrt , yolo	5	1846	December 18, 2023
Unexpected performance for INT8 ResNet50 on Jetson AGX Orin MAXN Jetson AGX Orin cuda	5	587	March 16, 2026
Error Code 10: Internal Error (Could not find any implementation for node PWN(/model.0/act/Sigmoid).) Jetson Orin Nano tensorrt	3	883	February 26, 2024
How can we know we have convert the onnx to int8trt rather than Float32? TensorRT tensorrt	23	2229	June 14, 2021
I do not get any performance improvement after using TensorRT provider for object detection model Jetson Nano tensorrt , onnx	6	1570	July 12, 2022

TensorRT Model Optimizer INT8 quantization causes 2.7x performance regression on Jetson Orin Nano 4GB (ViT-S + DPT architecture)