Description
Hi,
I’m using the TensorRT Python API to build a YOLOv8 model with the built-in NMS layer, support dynamic batch. NMS output1 is num detections, I added some slice layers to output0 to get batch indices, boxes, scores and classes. The model run fine, and I can handle output easily.
But when I deploy it to Triton Server and enable dynamic batching, the inference out becomes all zero. From what I understand, the built-in NMS output tensors are flattened, which make the first dimension of the output doesn’t match the first dimension of the input. Because Triton requires all model outputs to preserve the batch dimension for batching, this flattening seems to cause misalignment when multiple requests are merged dynamically.
Questions:
- Can I somehow handle or reshape the flattened NMS output so that it works with Triton dynamic batching?
- Or does the built-in NMS doesn’t work with Triton dynamic batching, meaning I need to handle batching myself before sending inference request to Triton?
TensorRT Version: 10.3.0
GPU Type: T4
Nvidia Driver Version: 550
CUDA Version: 12.4
CUDNN Version: 9.6.0
Operating System + Version: Ubuntu 22.04.5 LTS
Python Version: 3.10.12
Hi @hungthanh3123
The “all zero” output you are seeing is a classic signature of a Triton Scatter/Gather misalignment. This happens because the built-in NMS layer produces a “Ragged Tensor” (flattened list of detections), while Triton’s standard Dynamic Batcher expects a “Fixed Topology” (symmetrical outputs).
To answer your specific questions:
1. Can I somehow handle the flattened output with Triton Dynamic Batching? Technically Yes, but it requires “Decoupled Mode.” Standard Dynamic Batching enforces a 1-to-1 mapping: if the input batch has N images, the output tensor must be splittable into N equal slices. Your flattened NMS breaks this rule.
2. Does the built-in NMS not work with Triton Dynamic Batching? Correct, it does not work “out of the box” for dynamic batching. Because the built-in INMSLayer flattens the results into [Total_Detections, 7] and removes the batch structure, Triton attempts to slice the total detections by the batch size.
- Example: If Request A finds 5 objects and Request B finds 1 object (Batch=2, Total=6), Triton will blindly slice the output into two chunks of 3. Request A loses detections; Request B gets “ghost” detections from A. Often, this memory mismatch results in the zeroed buffers you observe.
Solution: Use Padding (EfficientNMS)
The industry-standard fix for YOLOv8 on Triton is to force the model to output a fixed shape so Triton can slice it cleanly.
Alternative: Client-Side Batching If you cannot change the model architecture, you must disable dynamic batching in Triton (max_batch_size: 0) and manually concatenate images on the client side into a single request. This bypasses Triton’s “scatter” logic entirely.
I hope this explains the root cause!
Is this solution applicable to TensorRT 10.12 and above, since EfficientNMS_TRT plugin has been deprecated?