# \#tensorrt-model-optimizer

**URL:** https://forums.developer.nvidia.com/tag/tensorrt-model-optimizer/935.md

[Latest](https://forums.developer.nvidia.com/latest.md) · [Categories](https://forums.developer.nvidia.com/categories.md) · [Tags](https://forums.developer.nvidia.com/tags.md)

---

## [Wrong FP16 results when a fused convolution, activation and residual add reads a strided concat view](https://forums.developer.nvidia.com/t/wrong-fp16-results-when-a-fused-convolution-activation-and-residual-add-reads-a-strided-concat-view/383599)

<div class="topic-metadata">

**Author:** [@Y-T-G](https://forums.developer.nvidia.com/u/Y-T-G)\
**Replies:** 0\
**Last updated:** [September 18, 2026, 9:17am UTC](https://forums.developer.nvidia.com/t/wrong-fp16-results-when-a-fused-convolution-activation-and-residual-add-reads-a-strided-concat-view/383599 "2026-09-18T09:17:16Z")

</div>

Description On TensorRT 11.2.1.2 a fused FP16 region of the form convolution, activation, residual add returns wrong values when the add operand is a non dense view of a later concatenation buffer. The fused layer reads…

---

## [Porting bit-exact QKV fusion from CPU to TensorRT](https://forums.developer.nvidia.com/t/porting-bit-exact-qkv-fusion-from-cpu-to-tensorrt/381122)

<div class="topic-metadata">

**Author:** [@argyrios-dev](https://forums.developer.nvidia.com/u/argyrios-dev)\
**Replies:** 0\
**Last updated:** [August 24, 2026, 12:47pm UTC](https://forums.developer.nvidia.com/t/porting-bit-exact-qkv-fusion-from-cpu-to-tensorrt/381122 "2026-08-24T12:47:53Z")

</div>

Hello, I developed a script that reduces an LLM’s RAM usage by over 8% using structural QKV fusion. A/B testing confirms the outputs are completely bit-exact, meaning there is zero knowledge loss. I am currently runnin…

---

## [TensorRT‑LLM fails on RTX PRO 2000 Blackwell with CUDA 13.2 (engine build + Python API + Triton backend)](https://forums.developer.nvidia.com/t/tensorrt-llm-fails-on-rtx-pro-2000-blackwell-with-cuda-13-2-engine-build-python-api-triton-backend/375541)

<div class="topic-metadata">

**Author:** [@vidjok](https://forums.developer.nvidia.com/u/vidjok)\
**Replies:** 1\
**Last updated:** [July 6, 2026, 12:11pm UTC](https://forums.developer.nvidia.com/t/tensorrt-llm-fails-on-rtx-pro-2000-blackwell-with-cuda-13-2-engine-build-python-api-triton-backend/375541 "2026-07-06T12:11:13Z")

</div>

Environment Overview GPU: NVIDIA RTX PRO 2000 Blackwell CUDA: 13.2 (required by driver 596.36) cuDNN: 9.x TensorRT: 10.x TensorRT‑LLM: latest public release OS: Windows 11 + WSL2 Ubuntu 22.04 Python:…

---

## [Real-Time Inference on Thor & RTX Pi0.5/GR00T N1.6/1.7 Thor 23 Hz RTX 5090 50-80Hz](https://forums.developer.nvidia.com/t/real-time-inference-on-thor-rtx-pi0-5-gr00t-n1-6-1-7-thor-23-hz-rtx-5090-50-80hz/368788)

<div class="topic-metadata">

**Author:** [@7thuniversels](https://forums.developer.nvidia.com/u/7thuniversels)\
**Replies:** 4\
**Last updated:** [May 6, 2026, 8:17am UTC](https://forums.developer.nvidia.com/t/real-time-inference-on-thor-rtx-pi0-5-gr00t-n1-6-1-7-thor-23-hz-rtx-5090-50-80hz/368788 "2026-05-06T08:17:46Z")

</div>

Hi everyone, I’m an independent developer with a background in algorithms, HPC, and robotics infrastructure. Recently I’ve been working on a lightweight inference engine built around hand-written CUDA kernels, focusing …

---

## [Jetson Thor - INT8 quantization show no performance gain over FP16 (2)](https://forums.developer.nvidia.com/t/jetson-thor-int8-quantization-show-no-performance-gain-over-fp16-2/360866)

<div class="topic-metadata">

**Author:** [@sopiti](https://forums.developer.nvidia.com/u/sopiti)\
**Replies:** 5\
**Last updated:** [March 27, 2026, 10:03am UTC](https://forums.developer.nvidia.com/t/jetson-thor-int8-quantization-show-no-performance-gain-over-fp16-2/360866 "2026-03-27T10:03:06Z")

</div>

Hello, I opened a post in december of last year regarding a lack of performance increase between FP32/FP16 and INT8/FP4 or a Thor ( Jetson Thor - INT8 quantization show no performance gain over FP16 ). Since then, the …

---

## [Nvfp4 Dynamic Quantizer Very Slow with Bias](https://forums.developer.nvidia.com/t/nvfp4-dynamic-quantizer-very-slow-with-bias/357007)

<div class="topic-metadata">

**Author:** [@quan.luo.101](https://forums.developer.nvidia.com/u/quan.luo.101)\
**Replies:** 3\
**Last updated:** [January 29, 2026, 9:06pm UTC](https://forums.developer.nvidia.com/t/nvfp4-dynamic-quantizer-very-slow-with-bias/357007 "2026-01-29T21:06:12Z")

</div>

Description I wrote a simple two layer GeMM and use model-optimizer to quantize it. Here’s the tensorrt profile results with both GeMMs bias=True in pytorch. Notice that the second Dyna layer is taking 1.84ms, which is …

---

## [Jetson Thor - INT8 quantization show no performance gain over FP16](https://forums.developer.nvidia.com/t/jetson-thor-int8-quantization-show-no-performance-gain-over-fp16/353308)

<div class="topic-metadata">

**Author:** [@sopiti](https://forums.developer.nvidia.com/u/sopiti)\
**Replies:** 7\
**Last updated:** [January 26, 2026, 10:02am UTC](https://forums.developer.nvidia.com/t/jetson-thor-int8-quantization-show-no-performance-gain-over-fp16/353308 "2026-01-26T10:02:10Z")

</div>

Hello, I’ve been playing with the Jetson Thor for a bit and I’ve gotten satisfying results for FP32/FP16 compared to the AGX’s. However, when it comes to INT8 or FP4, I notice that I am not seeing the expected performan…

---

## [Quantized GeMM using fp32 for Q/DQ layers](https://forums.developer.nvidia.com/t/quantized-gemm-using-fp32-for-q-dq-layers/356332)

<div class="topic-metadata">

**Author:** [@quan.luo.101](https://forums.developer.nvidia.com/u/quan.luo.101)\
**Replies:** 0\
**Last updated:** [January 2, 2026, 5:50pm UTC](https://forums.developer.nvidia.com/t/quantized-gemm-using-fp32-for-q-dq-layers/356332 "2026-01-02T17:50:19Z")

</div>

Description I’m using the nvcr.io/nvidia/pytorch:25.11-py3 docker to test nvfp4 on RTX-6000 Blackwell using Model-Optimizer. I wrote a simple two-layer GeMM and quantize it to nvfp4 using the \`torch\_quant\_to\_onnx.py\`. I…

---

## [Worse performance after quantization on TensorRT](https://forums.developer.nvidia.com/t/worse-performance-after-quantization-on-tensorrt/355549)

<div class="topic-metadata">

**Author:** [@Chachus](https://forums.developer.nvidia.com/u/Chachus)\
**Replies:** 1\
**Last updated:** [December 23, 2025, 1:19pm UTC](https://forums.developer.nvidia.com/t/worse-performance-after-quantization-on-tensorrt/355549 "2025-12-23T13:19:41Z")

</div>

Description Context: I am deploying the DEIMv2 model on a Jetson Orin NX (JetPack 6.2) and attempting to optimize performance using INT8 quantization via NVIDIA ModelOpt. The Issue: While my FP16/BF16 baseline achieves 5…

---

## [Failing to load pruned model yolov7 using tensorrt model opt](https://forums.developer.nvidia.com/t/failing-to-load-pruned-model-yolov7-using-tensorrt-model-opt/352560)

<div class="topic-metadata">

**Author:** [@abd.gang](https://forums.developer.nvidia.com/u/abd.gang)\
**Replies:** 1\
**Last updated:** [November 26, 2025, 5:23pm UTC](https://forums.developer.nvidia.com/t/failing-to-load-pruned-model-yolov7-using-tensorrt-model-opt/352560 "2025-11-26T17:23:21Z")

</div>

Hi, I am using Tensorrt MNodel Optimizer for pruning my yolov7 model. I used the below script to perform the pruning and it worked well. dummy\_input = torch.randn(1, 3, 640, 640).to(device) # Prune to a model with le…

---

## [TensorRT Quantization for Jetson Inference](https://forums.developer.nvidia.com/t/tensorrt-quantization-for-jetson-inference/347929)

<div class="topic-metadata">

**Author:** [@rictor](https://forums.developer.nvidia.com/u/rictor)\
**Replies:** 3\
**Last updated:** [October 17, 2025, 4:31am UTC](https://forums.developer.nvidia.com/t/tensorrt-quantization-for-jetson-inference/347929 "2025-10-17T04:31:48Z")

</div>

Hi Everyone, I’m currently working with a Jetson Orin Nano and I’m trying to optimize the model by using TensorRT. Right now the system has Jetpack 6.2 which means TensorRT is version 10.3. I managed to get the model eng…

---

## [ConvNeXT inference with int8 quantization slower on tensorRT than fp32/fp16](https://forums.developer.nvidia.com/t/convnext-inference-with-int8-quantization-slower-on-tensorrt-than-fp32-fp16/308761)

<div class="topic-metadata">

**Author:** [@Raj1234](https://forums.developer.nvidia.com/u/Raj1234)\
**Replies:** 2\
**Last updated:** [September 19, 2025, 10:05pm UTC](https://forums.developer.nvidia.com/t/convnext-inference-with-int8-quantization-slower-on-tensorrt-than-fp32-fp16/308761 "2025-09-19T22:05:17Z")

</div>

Description I am trying to quantize a convnext model to int8 but when I run inference it runs slower than my non quantized model. Environment TensorRT Version: 10.5.0 GPU Type: RTX 4090 Nvidia Driver Version: 556.12 C…

---

## [30% slowdown on ResNet50 with ModelOptimizer INT8 quantization (RTX 4090)](https://forums.developer.nvidia.com/t/30-slowdown-on-resnet50-with-modeloptimizer-int8-quantization-rtx-4090/345492)

<div class="topic-metadata">

**Author:** [@sai.prakash](https://forums.developer.nvidia.com/u/sai.prakash)\
**Replies:** 0\
**Last updated:** [September 19, 2025, 2:16pm UTC](https://forums.developer.nvidia.com/t/30-slowdown-on-resnet50-with-modeloptimizer-int8-quantization-rtx-4090/345492 "2025-09-19T14:16:29Z")

</div>

Hi all, I’m experimenting with explicit quantization using ModelOptimizer on a ResNet50 model. Setup: Started from PyTorch ResNet50 Exported ONNX with ModelOptimizer quantization following the PyTorch Quantizatio…

---

## [Holistically-Nested Edge Detection using TensoRT](https://forums.developer.nvidia.com/t/holistically-nested-edge-detection-using-tensort/315342)

<div class="topic-metadata">

**Author:** [@ariel27593](https://forums.developer.nvidia.com/u/ariel27593)\
**Replies:** 7\
**Last updated:** [March 18, 2025, 1:37pm UTC](https://forums.developer.nvidia.com/t/holistically-nested-edge-detection-using-tensort/315342 "2025-03-18T13:37:32Z")

</div>

Please provide complete information as applicable to your setup. • Hardware Platform Jetson Xavier • DeepStream Version 6.3.0 • JetPack Version 5.1.2 \*\*• TensorRT Version 8.5.2.2 \*\* • Issue Type:questions/bugs Hi, …

---

## [Errors with training flux of sparsity with accelerate](https://forums.developer.nvidia.com/t/errors-with-training-flux-of-sparsity-with-accelerate/325131)

<div class="topic-metadata">

**Author:** [@VieeoLee](https://forums.developer.nvidia.com/u/VieeoLee)\
**Replies:** 2\
**Last updated:** [March 4, 2025, 11:22am UTC](https://forums.developer.nvidia.com/t/errors-with-training-flux-of-sparsity-with-accelerate/325131 "2025-03-04T11:22:29Z")

</div>

Basic version info: python 3.12.0 pytorch 2.5.0 nvidia-modelopt 0.21.0 cuda: 12.6 I’m training flux-dev model of sparsity with accelerate. This is FSDP config with accelerator. distributed\_type: FSDP fsdp\_config: …

---

## [TensorRT examples](https://forums.developer.nvidia.com/t/tensorrt-examples/325109)

<div class="topic-metadata">

**Author:** [@siddharthaingermany](https://forums.developer.nvidia.com/u/siddharthaingermany)\
**Replies:** 1\
**Last updated:** [February 28, 2025, 1:09pm UTC](https://forums.developer.nvidia.com/t/tensorrt-examples/325109 "2025-02-28T13:09:20Z")

</div>

Description TensorFlow models are running very slow on Nvidia Jetson Orin AGX 64GB module. FaceNet model for face recognition is running at 2-3 FPS whereas the same model is showing around 10FPS in windows CPU laptop. I …

---

## [How to use same tensor rt version of Jetson orin nano in desktop PC environment](https://forums.developer.nvidia.com/t/how-to-use-same-tensor-rt-version-of-jetson-orin-nano-in-desktop-pc-environment/324824)

<div class="topic-metadata">

**Author:** [@dukhl89](https://forums.developer.nvidia.com/u/dukhl89)\
**Replies:** 3\
**Last updated:** [February 26, 2025, 8:39am UTC](https://forums.developer.nvidia.com/t/how-to-use-same-tensor-rt-version-of-jetson-orin-nano-in-desktop-pc-environment/324824 "2025-02-26T08:39:55Z")

</div>

As I know, Using converted tensor rt model is only support same tensor rt environment. (ex. If I converted the model by using tensor rt in Jetson orin nano system, This converted tensor rt model only can run in jetson or…

---

## [INT8 Calibration with DS 6.3 worse than with DS 6.0](https://forums.developer.nvidia.com/t/int8-calibration-with-ds-6-3-worse-than-with-ds-6-0/312135)

<div class="topic-metadata">

**Author:** [@gaylord](https://forums.developer.nvidia.com/u/gaylord)\
**Replies:** 19\
**Last updated:** [February 24, 2025, 8:04am UTC](https://forums.developer.nvidia.com/t/int8-calibration-with-ds-6-3-worse-than-with-ds-6-0/312135 "2025-02-24T08:04:45Z")

</div>

Please provide complete information as applicable to your setup. • Hardware Platform (Jetson / GPU) Jetson Orin NX (8GB) • DeepStream Version 6.3 • JetPack Version (valid for Jetson only) 5.1.2 • TensorRT Version …

---

## [Is there a plan to support DLA on the next TensorRT version?](https://forums.developer.nvidia.com/t/is-there-a-plan-to-support-dla-on-the-next-tensorrt-version/313130)

<div class="topic-metadata">

**Author:** [@chjej2021](https://forums.developer.nvidia.com/u/chjej2021)\
**Replies:** 4\
**Last updated:** [December 4, 2024, 6:30am UTC](https://forums.developer.nvidia.com/t/is-there-a-plan-to-support-dla-on-the-next-tensorrt-version/313130 "2024-12-04T06:30:45Z")

</div>

From TensorRT 10, “implicit quantization,” including IInt8Calibrator APIs, has been deprecated. In NVIDIA Jetson AGX Orin, DLA supports both FP16 and INT8 precision. To use INT8, implicit quantization is required. Also,…

---

## [\[TRT\] jetson agx orion error - CaffeParser: Could not open file device GPU, failed to load networks/Googlenet/bvlc\_googlenet.caffemodel](https://forums.developer.nvidia.com/t/trt-jetson-agx-orion-error-caffeparser-could-not-open-file-device-gpu-failed-to-load-networks-googlenet-bvlc-googlenet-caffemodel/308624)

<div class="topic-metadata">

**Author:** [@sbggopal](https://forums.developer.nvidia.com/u/sbggopal)\
**Replies:** 3\
**Last updated:** [October 18, 2024, 4:14am UTC](https://forums.developer.nvidia.com/t/trt-jetson-agx-orion-error-caffeparser-could-not-open-file-device-gpu-failed-to-load-networks-googlenet-bvlc-googlenet-caffemodel/308624 "2024-10-18T04:14:28Z")

</div>

Hi team, Am trying run inference of recognition program following the tutorial by buliding the repo, and i have installd all the utils and inference files , necessary to run the program. Also i have downloaded all the n…

---

## [Improving the speed for fp32 for yolov10x inference from Ultralytics on Jetson AGX Orin 64g devkit](https://forums.developer.nvidia.com/t/improving-the-speed-for-fp32-for-yolov10x-inference-from-ultralytics-on-jetson-agx-orin-64g-devkit/306786)

<div class="topic-metadata">

**Author:** [@mona.jalal](https://forums.developer.nvidia.com/u/mona.jalal)\
**Replies:** 4\
**Last updated:** [September 18, 2024, 7:23am UTC](https://forums.developer.nvidia.com/t/improving-the-speed-for-fp32-for-yolov10x-inference-from-ultralytics-on-jetson-agx-orin-64g-devkit/306786 "2024-09-18T07:23:44Z")

</div>

I have connected two RealSense D435 cameras to the Jetson AGX Orin 64g devkit (cameras show independent output and only detect the object). I was under the assumption that using int8 optimization for trt engine will yie…

---

## [Converting an ONNX model to TensorRT Engine on a x86/64 PC and then using it on a Jetson](https://forums.developer.nvidia.com/t/converting-an-onnx-model-to-tensorrt-engine-on-a-x86-64-pc-and-then-using-it-on-a-jetson/301511)

<div class="topic-metadata">

**Author:** [@neltherion](https://forums.developer.nvidia.com/u/neltherion)\
**Replies:** 2\
**Last updated:** [August 3, 2024, 7:16am UTC](https://forums.developer.nvidia.com/t/converting-an-onnx-model-to-tensorrt-engine-on-a-x86-64-pc-and-then-using-it-on-a-jetson/301511 "2024-08-03T07:16:42Z")

</div>

Is it possible to use a Workstation PC with an NVIDIA Graphics Card (Like GeForce 4090) to convert a large ONNX model into a TensorRT engine and then move the TensorRT engine to a Jetson device and run it there? I’m ask…

---

## [\[New\] Discord channel for triton-inference-server, tensorrt, tensorrt-llm, model-optimization](https://forums.developer.nvidia.com/t/new-discord-channel-for-triton-inference-server-tensorrt-tensorrt-llm-model-optimization/300055)

<div class="topic-metadata">

**Author:** [@Gerald10](https://forums.developer.nvidia.com/u/Gerald10)\
**Replies:** 0\
**Last updated:** [July 16, 2024, 8:15pm UTC](https://forums.developer.nvidia.com/t/new-discord-channel-for-triton-inference-server-tensorrt-tensorrt-llm-model-optimization/300055 "2024-07-16T20:15:39Z")

</div>

Hi, I noticed there is no slack, discord or irc channel for tensorrt - which could offload some future tickets by discussing things in the channel and getting instant help - so I created one. I hope its ok to advertise…

---

## [TensorRT 10.2 is not using FP8 convolution tactics when building a FP8 quantized conv model](https://forums.developer.nvidia.com/t/tensorrt-10-2-is-not-using-fp8-convolution-tactics-when-building-a-fp8-quantized-conv-model/298985)

<div class="topic-metadata">

**Author:** [@pord74571](https://forums.developer.nvidia.com/u/pord74571)\
**Replies:** 2\
**Last updated:** [July 10, 2024, 6:19pm UTC](https://forums.developer.nvidia.com/t/tensorrt-10-2-is-not-using-fp8-convolution-tactics-when-building-a-fp8-quantized-conv-model/298985 "2024-07-10T18:19:17Z")

</div>

Description TensorRT 10.2 added support for normal FP8 convolutions on Hopper GPUs. So, I built an engine with a simple QDQ+Conv2d model quantized by TensorRT-Model-Optimizer, but, no FP8 Conv tactics are timed. How…

---

## [GPUs hang when executing NIM docker container on a 4xA100](https://forums.developer.nvidia.com/t/gpus-hang-when-executing-nim-docker-container-on-a-4xa100/296880)

<div class="topic-metadata">

**Author:** [@yirunw](https://forums.developer.nvidia.com/u/yirunw)\
**Replies:** 2\
**Last updated:** [June 29, 2024, 10:09pm UTC](https://forums.developer.nvidia.com/t/gpus-hang-when-executing-nim-docker-container-on-a-4xa100/296880 "2024-06-29T22:09:46Z")

</div>

Description When executing the NIM docker container with the following script, if only one gpu is available or a gpu is specified, it runs normally. If multiple gpus are available, it will hang and the gpu usage will be …

---

## [/TopK\_5: K exceeds the maximum value allowed (3840)](https://forums.developer.nvidia.com/t/topk-5-k-exceeds-the-maximum-value-allowed-3840/295903)

<div class="topic-metadata">

**Author:** [@jack.everson](https://forums.developer.nvidia.com/u/jack.everson)\
**Replies:** 0\
**Last updated:** [June 11, 2024, 1:04am UTC](https://forums.developer.nvidia.com/t/topk-5-k-exceeds-the-maximum-value-allowed-3840/295903 "2024-06-11T01:04:04Z")

</div>

Description I have been having issues with getting a .onnx model to build using either TensorRT or trtexec.exe. I receive the following error using TRT in C++ on the call: std::unique\_ptr\<IHostMemory\> plan{ builder-\>bui…
