Using yolo detection from deepstream for counting & tracking

How can we use the detections made through yolov3 on deepstream 4.0.1 for things like counting and tracking?

There’s some mention of plugins, but I haven’t found something concrete for this. How can we build custom
code for different things and implement it, after detections are made using deepstream.

I’m using the Jetson nano.

Hi,

How can we use the detections made through yolov3 on deepstream 4.0.1 for things like counting and tracking?

If you want to count the objects or add any custom logic based on counting, you can have a look at the test1-app in the sdk. The counting can be done in osd_sink_pad_buffer_probe. The sample uses the restnet model by default. You can switch to yolo model by replacing the config file.

For tracking, do you mean object tracking ? If yes, we already have implementations for object tracker available. You can refer to documentation for further details - https://docs.nvidia.com/metropolis/index.html

Tracker documentation - https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream_Plugin_Manual%2Fdeepstream_plugin_details.02.02.html

Hi @NvCJR, thanks for replying.

Yes - object tracking indeed. I have tried the various tracking methods mentioned in the docs. While it’s good that they are there, but the bounding boxes flicker way too much and that makes the overall count unusable.

Essentially, what i need is a way to access the bounding boxes from the tracker plugin output so I can augment them in OpenCV in a custom file. In this file I’ll be able to do things like finding the coordinates, centroids etc.

How can I do this? And then pass the output of my custom file back to the nvdosd plugin (on screen display) ?

I have also looked at the dsExample in the docs, but the docs for this are hard to follow for my use case.

While it’s good that they are there, but the bounding boxes flicker way too much and that makes the overall count unusable.

Do you see this behavior on sample streams provided with the sdk or is this specific to your own data ? Tracker parameters often need to be tuned based on the specific use case you are trying to solve. What trackers did you try using ?

so I can augment them in OpenCV in a custom file

Could you please explain this in more detail ? Do you only need bounding box information ? If yes, you can do that in the “osd_sink_pad_buffer_probe” function from test1 app. Once you have done your computations, you can update the same metadata obtained from GstBuffer. This is being read at the input of OSD plugin so any changes you make would be taken into account by OSD plugin.

Do you see this behavior on sample streams provided with the sdk or is this specific to your own data ?

We saw this with both the sample streams and our custom files as well. A lot of flickering in both, even when we used the NvDCF tracker.

Could you please explain this in more detail ? Do you only need bounding box information ?

Well as a first step, I want to stabilise the bounding boxes so I was going to track the coordinates of the bbox with my custom logic.
Then I have to take this further, e.g. estimate which direction objects are moving, see when they cross a line and other custom analysis.

So really, having knowledge of how I can take the output from the tracker plugin, use it in my custom opencv file/plugin and then pass the output from this custom opencv file back to the nvdosd plugin (display) is what I need. (because I want to do things that are outside the scope of the current functions as well).

Here’s a diagram explaining it in terms of the plugin sequence http://tiny.cc/di2iez

We saw this with both the sample streams and our custom files as well. A lot of flickering in both, even when we used the NvDCF tracker.

Did you make any changes to the tracker settings when you tried it with the sample video from the SDK ? If yes, can you share the application and tracker config files you have used where you see flickering?

For your second question, there are two approaches you can try -

  1. As i mentioned earlier, use the osd_sink_pad_buffer_probe function to parse the metadata and then make changes based on your logic. This is easy to implement since most the code in already present in test1-app.

  2. Write your own custom plugin which would do parse the metadata and then make changes based on your logic. You can refer to DS Example plugin for this approach. You can refer how to parse/update metadata from that plugin source code. The source code has been annotated with comments to help understand what’s being done.

Hi @imbatraman I am at the same step like you, did you solve your question?
I am trying to read about Gst-nvtracker for count all detections.
thanks for your time.