Poor performance due to INMSLayers followup nms_layer_output(1)[DevicetoShapeHostCopy] and trainstation

johannesrhvw · September 27, 2025, 8:18am

After looking a bit more, it seems this has been an issue for a longer time and there is never an answer to this topic from nvidia.

github.com/NVIDIA/TensorRT

Non-Maximal-Suppression (NMS) Layers slow on TensorRT 10.0-10.6

opened 10:00PM - 14 Nov 24 UTC

darrin-willis

Module:Performance triaged

## Description NMS Layers are much slower on TensorRT than on PyTorch (44% of t…he performance) and I'm looking for any possible workaround. This seems to be acknowledged as a known issue in the TensorRT release notes [here](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#rel-10-6-0): > A performance regression is expected for TensorRT 10.x with respect to TensorRT 8.6 for networks with operations that involve data-dependent shapes, such as non-max suppression or non-zero operations Is there any possible workaround or a fix planned in a specific future version? I am specifically using these layers inside a `FasterRCNN` network (as implemented in `torchvision` [here](https://pytorch.org/vision/main/models/faster_rcnn.html)). I observe this network to be much slower when running either with a single image or 4 images: - Single image inference latency: 7.8ms on PyTorch, 13.3ms on TensorRT - 4 image inference latency: 22.8ms on PyTorch, 53.5ms on TensorRT When I run this network with per-layer profiling, I see that the `NonMaxSuppression` layers account for 75%+ of the overall inference time. I have verified this on TensorRT 10.0 and 10.6. I have tested using ONNX opset 11 and opset 17. ## Environment **TensorRT Version**: 10.0, 10.6 **NVIDIA GPU**: GeForce RTX 4090 **NVIDIA Driver Version**: 550.54.15 **CUDA Version**: 12.4 **CUDNN Version**: unsure Operating System: Python Version (if applicable): 3.9 Tensorflow Version (if applicable): PyTorch Version (if applicable): 2.2 Baremetal or Container (if so, version): ## Relevant Files **Model link**: https://pytorch.org/vision/main/models/faster_rcnn.html ## Steps To Reproduce 1. Export FasterRCNN to ONNX 2. Pass ONNX into `trtexec` 4. Compare `trtexec` output to PyTorch equivalent benchmark **Commands or scripts**: **Have you tried [the latest release](https://developer.nvidia.com/tensorrt)?**: Yes I have tried TensorRT 10.6 and 10.0 **Can this model run on other frameworks?** For example run ONNX model with ONNXRuntime (`polygraphy run <model.onnx> --onnxrt`): Yes it runs on onnxruntime.

as stated here and in my previous post Output-tensor-meta Access RAW model output with batch dimension i will keep going with NMS and RoiAlign outside the model with a custom postprocessor, until someone feels obligated to give an answer this issue.

Topic		Replies	Views
Confusing NMS layer implementation in TensorRT 8.5 TensorRT tensorrt , onnx	2	1404	May 8, 2024
How to use INMSLayer with DeepStream 8.0 when dynamic sizes are not supported? DeepStream SDK tensorrt , machine-learning , deepstream	5	112	November 18, 2025
TensorRT built-in NMS output lost when using Triton dynamic batching TensorRT tensorrt , cudnn , inference-server-triton	3	143	December 16, 2025
PyTorch usage of INMSLayer on TensorRT TensorRT pytorch	3	876	January 4, 2025
INMSLayer: CUDA graph invalidation (DevicetoShapeHostCopy) TensorRT cuda	4	142	September 1, 2025
TensorRT ONNX NonMaxSuppression outputs only 1 index (expected many) TensorRT	0	26	February 3, 2026
Output-tensor-meta Access RAW model output with batch dimension DeepStream SDK cuda , deepstream	9	153	September 25, 2025
Make tiny yolo v4 onnx output compatible with nms layer batchedNMSPlugin (C++/TensoRT) TensorRT	2	1633	November 24, 2021
Down performance when add layer to TensorRT convert from ONNX TensorRT	2	448	June 19, 2020
Writing layer for NonMaxSuppression in onnx parser DRIVE AGX Xavier General driveos-dl	21	4061	October 12, 2021

Poor performance due to INMSLayers followup nms_layer_output(1)[DevicetoShapeHostCopy] and trainstation

Related topics