Class-agnostic NMS plugin

tadej.svetina · December 16, 2022, 8:03pm

What

It would be great if you could include an option to make the BatchedNMS plugin perform NMS filtering in a class-agnostic way. Right now, the plugin only filters based on IoU scores among detections belonging to the same class - what I would like is an option to do this across all detections, regardless of the class.

Why

In my case, the detector has trouble distinguishing between two classes. That’s why it can produce detections only a few pixels apart, with some having the highest score for class A, and others for class B. The class-wise way of NMS filtering will unfortunately then return two such almost identical detection boxes, except that one has class A, the other class B - although they are both representing the same object.

In my dataset I found that this actually significantly hurts the precision scores of affected classes.

How

The best way would be to add classAgnostic or similar attribute to the plugin. I’ve tried everything I could think of - it’s not possible to achieve what I want with the current plugin (giving all instances the same class means that there is no way to recover the actual class). I would be glad to help with this, but I have very limited experince with C++ and CUDA, and not that much time :(

This behavior is supported by ONNX NMS operator, which also has limited support in TensorRT (but does not support dynamic shapes, which is why I can’t use it).

NVES · December 16, 2022, 8:37pm

Hi,
Please refer to below links related custom plugin implementation and sample:

While IPluginV2 and IPluginV2Ext interfaces are still supported for backward compatibility with TensorRT 5.1 and 6.0.x respectively, however, we recommend that you write new plugins or refactor existing ones to target the IPluginV2DynamicExt or IPluginV2IOExt interfaces instead.

Thanks!

tadej.svetina · December 16, 2022, 8:40pm

Thanks, but what I am suggesting is a change to the existing plugin

tadej.svetina · December 17, 2022, 9:59am

I see that EfficientNMS plug in has the same problem.

spolisetty · December 21, 2022, 9:47am

Hi,

The EfficientNMS plugin will support class-agnostic filtering in its coming updates. If you are using BatchedNMS currently, it should be easy to switch to EfficientNMS as both plugins share similar input/output tensor specs.

In the meantime, if the you needs this change urgently (and if you are comfortable with editing and recompiling plugin source code) you can do this change easily by removing the following line in the code and rebuilding the plugin library.

threadClass[tile] == testClass &&

Removing that check in the if statement will skip the class-specific filtering, and the plugin would then effectively operate as class-agnostic instead.

github.com

NVIDIA/TensorRT/blob/main/plugin/efficientNMSPlugin/efficientNMSInference.cu#L322


      
                  MapNMSData<T, Tb>(param, i, imageIdx, boxesInput, anchorsInput, topClassData, topAnchorsData, topNumData,
                      sortedScoresData, sortedIndexData, testScore, testClass, testBox, testBoxIdxMap);
          
          
        for (int tile = 0; tile < numTiles; tile++)
                  {
                      // IOU
                      if (boxIdx[tile] > i && // Make sure two different boxes are being tested, and that it's a higher index;
                          boxIdx[tile] < numSelectedBoxes && // Make sure the box is within numSelectedBoxes;
                          blockState == 1 &&                 // Signal that allows IOU checks to be performed;
                          threadState[tile] == 0 &&          // Make sure this box hasn't been either dropped or kept already;
                          threadClass[tile] == testClass &&  // Compare only boxes of matching classes;
                          lte_mp(threadScore[tile], testScore) && // Make sure the sorting order of scores is as expected;
                          IOU<T>(param, threadBox[tile], testBox) >= param.iouThreshold) // And... IOU overlap.
                      {
                          // Current box overlaps with the box tested in this iteration, this box will be skipped.
                          threadState[tile] = -1; // -1 => Mark this box's thread to be dropped.
                      }
                  }
              }
          }

Thank you.

tadej.svetina · December 21, 2022, 7:29pm

@spolisetty Great news, thank you! I have actually already switched to EfficientNMS plugin, as it has a simpler interface and was a bit faster.

Can I ask when can we expect the new version of the plugin with agnostic NMS to be released?

spolisetty · December 26, 2022, 8:01am

Currently we are not sure about ETA, we can expect it in coming couple of weeks.

Thank you.

system · January 9, 2023, 8:01am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Efficient NMS plugin to TensorRT engine at runtime TensorRT	4	6713	May 17, 2022
Onnx to tensorrt plugin for NonMaxSuppression TensorRT tensorrt , tensorflow	1	2500	April 26, 2020
Confusing NMS layer implementation in TensorRT 8.5 TensorRT tensorrt , onnx	2	1203	May 8, 2024
Onnx2TRT missing TensorListStack plugin TensorRT	1	1085	December 9, 2021
ONNX Plugin Layer implements TensorRT	11	1913	January 12, 2021
Adding BatchedNMSDynamic_TRT plugin in the ssd mobileNet onnx model TensorRT	4	1876	September 7, 2021
Embed EfficientNMS_TRT in Detectnet_v2? TAO Toolkit	4	284	January 27, 2024
Writing layer for NonMaxSuppression in onnx parser DRIVE AGX Xavier General driveos-dl	21	3752	October 12, 2021
Using Tensorrt Batched NMS Plugin with SCRFD Model - Insightface TensorRT	1	785	June 14, 2023
Use pre-trained object detection TF2 models with TensorRT ONNX TensorRT	9	1930	May 31, 2021

Class-agnostic NMS plugin

What

Why

How

Related topics