Feasibility of Using IOU and NvDCF Tracker with Instance Segmentation

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version since 6.2
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hello

I’m exploring the capabilities of Instance Segmentation in combination with tracking algorithms and have a specific question regarding its compatibility with the IOU and NvDCF tracker or DeepSort tracker

In my previous experiences, I’ve successfully employed trackers with detection-based models. However, I’m curious about the potential integration of these trackers with an instance segmentation approach. Specifically, I understand that instance segmentation can automatically generate bounding boxes (rect_param) based on minimum and maximum parameters. Given this, I wonder if it’s feasible to link a tracker, such as the NvDCF, to an instance segmentation detector and achieve effective tracking.

Has anyone experimented with this setup or can provide insights into whether the NvDCF tracker works reliably with instance segmentation outputs? Any guidance or shared experiences would be greatly appreciated.

Thank you in advance for your help!

3

Thanks for your suggestion. Let me check internally and feedback.

4

Oh, can we say that only detectors are supported with tracker ?

5

Yes, nvtracker need bboxes from detector to implement tracking.

6

To provide more information, current sample pipelines use detector + tracker, but your idea is feasible. It’s possible to get the minimum and maximum from instance segmentation to create a bounding box and feed it to tracker. You can implement a customized instance segmentation model as PGIE.

7

Then, even if it is applied, if interval > 0, bbox tracking will be possible, but mask tracking within it will be difficult.
So, I think this could be possible by combining nano-sam developed by nvidia. I hope the function will be developed.