# TensorRT built-in NMS output lost when using Triton dynamic batching **URL:** \ **Category:** TensorRT\ **Tags:** inference-server-triton, tensorrt, cudnn\ **Created:** [November 3, 2025, 7:42am UTC](https://forums.developer.nvidia.com/t/tensorrt-built-in-nms-output-lost-when-using-triton-dynamic-batching/350004 "2025-11-03T07:42:36Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![hungthanh3123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@hungthanh3123](https://forums.developer.nvidia.com/u/hungthanh3123)\ **Post date:** [November 3, 2025, 7:42am UTC](https://forums.developer.nvidia.com/t/tensorrt-built-in-nms-output-lost-when-using-triton-dynamic-batching/350004/1 "2025-11-03T07:42:36Z")
## Description Hi, I’m using the TensorRT Python API to build a YOLOv8 model with the built-in NMS layer, support dynamic batch. NMS output1 is num detections, I added some slice layers to output0 to get batch indices, boxes, scores and classes. The model run fine, and I can handle output easily. But when I deploy it to Triton Server and enable dynamic batching, the inference out becomes all zero. From what I understand, the built-in NMS output tensors are flattened, which make the first dimension of the output doesn’t match the first dimension of the input. Because Triton requires all model outputs to preserve the batch dimension for batching, this flattening seems to cause misalignment when multiple requests are merged dynamically. Questions: 1. Can I somehow handle or reshape the flattened NMS output so that it works with Triton dynamic batching? 2. Or does the built-in NMS doesn’t work with Triton dynamic batching, meaning I need to handle batching myself before sending inference request to Triton? **TensorRT Version** : 10.3.0 **GPU Type** : T4 **Nvidia Driver Version** : 550 **CUDA Version** : 12.4 **CUDNN Version** : 9.6.0 **Operating System + Version** : Ubuntu 22.04.5 LTS **Python Version** : 3.10.12 ---
**Author:** ![athkumar](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/athkumar/32/382649_2.png) [@athkumar](https://forums.developer.nvidia.com/u/athkumar)\ **Post date:** [November 27, 2025, 9:16am UTC](https://forums.developer.nvidia.com/t/tensorrt-built-in-nms-output-lost-when-using-triton-dynamic-batching/350004/4 "2025-11-27T09:16:57Z")
Hi @hungthanh3123 The “all zero” output you are seeing is a classic signature of a **Triton Scatter/Gather misalignment**. This happens because the built-in NMS layer produces a “Ragged Tensor” (flattened list of detections), while Triton’s standard Dynamic Batcher expects a “Fixed Topology” (symmetrical outputs). To answer your specific questions: **1. Can I somehow handle the flattened output with Triton Dynamic Batching?** **Technically Yes, but it requires “Decoupled Mode.”** Standard Dynamic Batching enforces a 1-to-1 mapping: if the input batch has `N` images, the output tensor must be splittable into `N` equal slices. Your flattened NMS breaks this rule. **2. Does the built-in NMS not work with Triton Dynamic Batching?** **Correct, it does not work “out of the box” for dynamic batching.** Because the built-in `INMSLayer` flattens the results into `[Total_Detections, 7]` and removes the batch structure, Triton attempts to slice the total detections by the batch size. - _Example:_ If Request A finds 5 objects and Request B finds 1 object (Batch=2, Total=6), Triton will blindly slice the output into two chunks of 3. Request A loses detections; Request B gets “ghost” detections from A. Often, this memory mismatch results in the zeroed buffers you observe. ### Solution: Use Padding (EfficientNMS) The industry-standard fix for YOLOv8 on Triton is to force the model to output a **fixed shape** so Triton can slice it cleanly. - **Plugin:** Use the `EfficientNMS_TRT` plugin (standard in YOLO export scripts). Refer: [YOLOv11 to ONNX (with NMS and dynamic batch) · Issue #18172 · ultralytics/ultralytics · GitHub](https://github.com/ultralytics/ultralytics/issues/18172) - **Output Shape:** Configure it to output `[Batch_Size, Max_Boxes, 6]` (e.g., `[B, 300, 6]`). - **Padding:** The layer will pad valid detections with zeros to fill the `300` slots. **Alternative: Client-Side Batching** If you cannot change the model architecture, you must disable dynamic batching in Triton (`max_batch_size: 0`) and manually concatenate images on the client side into a single request. This bypasses Triton’s “scatter” logic entirely. I hope this explains the root cause! ---
**Author:** ![victoria14](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@victoria14](https://forums.developer.nvidia.com/u/victoria14)\ **Post date:** [December 2, 2025, 8:42pm UTC](https://forums.developer.nvidia.com/t/tensorrt-built-in-nms-output-lost-when-using-triton-dynamic-batching/350004/5 "2025-12-02T20:42:40Z")
Is this solution applicable to TensorRT 10.12 and above, since `EfficientNMS_TRT` plugin has been deprecated? ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [December 16, 2025, 8:43pm UTC](https://forums.developer.nvidia.com/t/tensorrt-built-in-nms-output-lost-when-using-triton-dynamic-batching/350004/6 "2025-12-16T20:43:15Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.