Jetson Thor AGX - Poor INT8 performance

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

Problem

The Jetson Thor AGX shows no performance improvement when running models in INT8/best vs FP16. On the Orin AGX we see +37% uplift in performance, also similar on dGPUs.

This ticket has the same issue, but has been closed without a follow up/resolution. This is still a problem, we’ve spent a heaps of money on this Thor and I’d expect it to perform better.

Is there something we are doing wrong, or a fix incoming?

I’ve only tested on batch size 1 and i notice you say I should increase the batch size, but I’d still expect some increase.

Steps to reproduce

Environment

Thor AGX

# R38 (release), REVISION: 4.0, GCID: 43443517, BOARD: generic, EABI: aarch64, DATE: Wed Dec 31 00:15:19 UTC 2025
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
INSTALL_TYPE=

docker run -it --rm --runtime nvidia --network host -v ~/models:/models nvcr.io/nvidia/deepstream:8.0-triton-multiarch bash

Orin AGX

# R36 (release), REVISION: 4.0, GCID: 37537400, BOARD: generic, EABI: aarch64, DATE: Fri Sep 13 04:36:44 UTC 2024
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia

docker run -it --rm --runtime nvidia --network host -v ~/models:/models nvcr.io/nvidia/deepstream:7.1-triton-multiarch bash

Power mode: MAX_N
Model: rtdetr2

Run benchmark

fp16

trtexec --onnx=/models/rtdetrv2.onnx --fp16 --shapes=images:1x3x1280x1280 --saveEngine=/models/rtdetrv2.fp16.engine

int8

trtexec --onnx=/models/rtdetrv2.onnx --best --shapes=images:1x3x1280x1280 --saveEngine=/models/rtdetrv2.best.engine --calib=calibration.cache

Benchmark results

Thor AGX

fp16

[03/18/2026-06:27:16] [I] === Performance summary ===                                                                          
[03/18/2026-06:27:16] [I] Throughput: 156.263 qps                                                                                                                                                                                                             
[03/18/2026-06:27:16] [I] Latency: min = 6.02441 ms, max = 7.21313 ms, mean = 6.46305 ms, median = 6.31262 ms, percentile(90%) = 7.00299 ms, percentile(95%) = 7.06018 ms, percentile(99%) = 7.16968 ms                                                       
[03/18/2026-06:27:16] [I] Enqueue Time: min = 1.1311 ms, max = 1.82593 ms, mean = 1.29942 ms, median = 1.21533 ms, percentile(90%) = 1.66211 ms, percentile(95%) = 1.74329 ms, percentile(99%) = 1.8131 ms                                                    
[03/18/2026-06:27:16] [I] H2D Latency: min = 0.129883 ms, max = 0.572693 ms, mean = 0.20111 ms, median = 0.176514 ms, percentile(90%) = 0.297119 ms, percentile(95%) = 0.352295 ms, percentile(99%) = 0.477783 ms                                             
[03/18/2026-06:27:16] [I] GPU Compute Time: min = 5.85181 ms, max = 7.01965 ms, mean = 6.24976 ms, median = 6.12573 ms, percentile(90%) = 6.76337 ms, percentile(95%) = 6.82196 ms, percentile(99%) = 6.90288 ms                                              
[03/18/2026-06:27:16] [I] D2H Latency: min = 0.00488281 ms, max = 0.0592651 ms, mean = 0.0121775 ms, median = 0.0136719 ms, percentile(90%) = 0.0217285 ms, percentile(95%) = 0.0228577 ms, percentile(99%) = 0.0335083 ms                                    
[03/18/2026-06:27:16] [I] Total Host Walltime: 3.01414 s                                                                       
[03/18/2026-06:27:16] [I] Total GPU Compute Time: 2.94364 s

best

[03/18/2026-07:11:58] [I] === Performance summary ===
[03/18/2026-07:11:58] [I] Throughput: 160.521 qps
[03/18/2026-07:11:58] [I] Latency: min = 5.9176 ms, max = 7.00183 ms, mean = 6.24156 ms, median = 6.15625 ms, percentile(90%) = 6.65479 ms, percentile(95%) = 6.73944 ms, percentile(99%) = 6.81396 ms
[03/18/2026-07:11:58] [I] Enqueue Time: min = 1.39868 ms, max = 2.02954 ms, mean = 1.65265 ms, median = 1.57593 ms, percentile(90%) = 1.93433 ms, percentile(95%) = 1.98135 ms, percentile(99%) = 2.00287 ms
[03/18/2026-07:11:58] [I] H2D Latency: min = 0.136719 ms, max = 0.251465 ms, mean = 0.172315 ms, median = 0.172119 ms, percentile(90%) = 0.200867 ms, percentile(95%) = 0.206055 ms, percentile(99%) = 0.224365 ms
[03/18/2026-07:11:58] [I] GPU Compute Time: min = 5.75623 ms, max = 6.7605 ms, mean = 6.05089 ms, median = 5.97205 ms, percentile(90%) = 6.44641 ms, percentile(95%) = 6.50885 ms, percentile(99%) = 6.57623 ms
[03/18/2026-07:11:58] [I] D2H Latency: min = 0.00488281 ms, max = 0.106934 ms, mean = 0.0183527 ms, median = 0.020874 ms, percentile(90%) = 0.0339355 ms, percentile(95%) = 0.0379639 ms, percentile(99%) = 0.059082 ms
[03/18/2026-07:11:58] [I] Total Host Walltime: 3.00895 s
[03/18/2026-07:11:58] [I] Total GPU Compute Time: 2.92258 s

Orin AGX

fp16

[03/18/2026-07:02:41] [I] === Performance summary ===
[03/18/2026-07:02:41] [I] Throughput: 44.3501 qps
[03/18/2026-07:02:41] [I] Latency: min = 22.5975 ms, max = 23.8183 ms, mean = 23.2262 ms, median = 23.278 ms, percentile(90%) = 23.5813 ms, percentile(95%) = 23.6375 ms, percentile(99%) = 23.7903 ms
[03/18/2026-07:02:41] [I] Enqueue Time: min = 1.93848 ms, max = 3.94531 ms, mean = 2.96401 ms, median = 2.94235 ms, percentile(90%) = 3.10669 ms, percentile(95%) = 3.17078 ms, percentile(99%) = 3.94385 ms
[03/18/2026-07:02:41] [I] H2D Latency: min = 0.81189 ms, max = 0.865479 ms, mean = 0.834716 ms, median = 0.834137 ms, percentile(90%) = 0.844482 ms, percentile(95%) = 0.846558 ms, percentile(99%) = 0.861328 ms
[03/18/2026-07:02:41] [I] GPU Compute Time: min = 21.7526 ms, max = 22.9693 ms, mean = 22.3795 ms, median = 22.4308 ms, percentile(90%) = 22.7393 ms, percentile(95%) = 22.8079 ms, percentile(99%) = 22.9446 ms
[03/18/2026-07:02:41] [I] D2H Latency: min = 0.00537109 ms, max = 0.0197144 ms, mean = 0.0120598 ms, median = 0.0114746 ms, percentile(90%) = 0.0168457 ms, percentile(95%) = 0.0184631 ms, percentile(99%) = 0.0192871 ms
[03/18/2026-07:02:41] [I] Total Host Walltime: 3.06651 s
[03/18/2026-07:02:41] [I] Total GPU Compute Time: 3.04361 s

best

[03/18/2026-06:40:27] [I] === Performance summary ===
[03/18/2026-06:40:27] [I] Throughput: 60.5375 qps
[03/18/2026-06:40:27] [I] Latency: min = 16.9979 ms, max = 17.4189 ms, mean = 17.177 ms, median = 17.1806 ms, percentile(90%) = 17.2783 ms, percentile(95%) = 17.3104 ms, percentile(99%) = 17.3787 ms
[03/18/2026-06:40:27] [I] Enqueue Time: min = 2.50696 ms, max = 2.69939 ms, mean = 2.57615 ms, median = 2.57788 ms, percentile(90%) = 2.6228 ms, percentile(95%) = 2.65172 ms, percentile(99%) = 2.67404 ms
[03/18/2026-06:40:27] [I] H2D Latency: min = 0.720276 ms, max = 0.746216 ms, mean = 0.732666 ms, median = 0.732544 ms, percentile(90%) = 0.737915 ms, percentile(95%) = 0.739014 ms, percentile(99%) = 0.743164 ms
[03/18/2026-06:40:27] [I] GPU Compute Time: min = 16.2415 ms, max = 16.6697 ms, mean = 16.428 ms, median = 16.4308 ms, percentile(90%) = 16.5293 ms, percentile(95%) = 16.5634 ms, percentile(99%) = 16.6225 ms
[03/18/2026-06:40:27] [I] D2H Latency: min = 0.00537109 ms, max = 0.022522 ms, mean = 0.0163163 ms, median = 0.0164795 ms, percentile(90%) = 0.0209961 ms, percentile(95%) = 0.0215302 ms, percentile(99%) = 0.0222168 ms
[03/18/2026-06:40:27] [I] Total Host Walltime: 3.05596 s
[03/18/2026-06:40:27] [I] Total GPU Compute Time: 3.03918 s
3

Hi,

Below is a similar topic for your reference:

Could you test with --dumpLayerInfo --dumpProfile --profilingVerbosity=detailed --separateProfileRun --useCudaGraph --noDataTransfers argument and share the output with us?

Thanks.

4

Thanks for your quick reply.

Here’s trtexec log for int8 on the Thor, with the args requested:

trt.log (204.6 KB)

5

Hi,

For the most time-consuming layer:

[03/18/2026-23:12:54] [I] === Profile (487 iterations ) ===
[03/18/2026-23:12:54] [I]    Time(ms)     Avg.(ms)   Median(ms)   Time(%)   Layer
[03/18/2026-23:12:54] [I]       49.41       0.1015       0.1014       1.6   __myl_MulMinMaxRounCastConc_myl0_0

The node (kgen) comes from the Myelin compiler:

Name: __myl_MulMinMaxRounCastConc_myl0_0, LayerType: kgen, Inputs: [ { Name: images, Dimensions: [1,3,1280,1280], Format/Datatype: Float }], Outputs: [ { Name: /model/backbone/conv1/conv1_1/act/Relu_output_0 Q output.1_2_myl0, Dimensions: [1,4,1280,1280], Format/Datatype: Int8 }], TacticName: __myl_MulMinMaxRounCastConc_0x8d78bf2c7f72037dec07a3eb519b4387, StreamId: 0, Metadata: [ONNX Layer: /model/backbone/conv1/conv1_1/conv/Conv][ONNX Layer: /model/backbone/conv1/conv1_1/act/Relu]

To check it further, could you share the rtdetrv2.rgb.v5.champion.onnx model with us?
Thanks.

6

I’ve sent you a download link to the .onnx with random weights.

Let me know if there’s something I can do on my end.

Thanks!

7

Hi,

Thanks for sharing the model.
Confirmed that we can reproduce a similar behavior in our internal device.
We need to check with our internal and get back to you later.

Thanks.

9

Hi,

Please try to use modelopt with explicit quant.

Step 1: Install ModelOpt

pip install "nvidia-modelopt[onnx]"

Step 2: Quantize the ONNX model

python3 -m modelopt.onnx.quantization \
    --quantize_mode=int8 \
    --onnx_path=rtdetrv2.rgb.v5.champion.random.onnx \
    --output_path=rtdetrv2.INT8.onnx

Step 3: Run trtexec with --stronglyTyped

/usr/src/tensorrt/bin/trtexec \
    --onnx=rtdetrv2.INT8.onnx \
    --stronglyTyped \
    --shapes=images:1x3x1280x1280 \
    --useCudaGraph --noDataTransfers

Thanks.