DRIVE 9 Clusterer & Tracker Help

We are working with DRIVE 9. The detections are pretty good but we are struggling with the tracker. Please take a look at https://imgur.com/6NdGLf6 to view the problem.

The tracker (or perhaps clusterer?) seems to be merging the boxes of the two depicted pedestrians. To produce this image, I reset all parameter settings to those found in the sample_object_detector_tracker sample from DRIVE 9. Note, however, that we’re not using the DNN from that example to do the detections; we’re using DriveNet. Also note that we’re not doing our own clustering but using the 2D box tracker’s internal clustering.

My next avenue of experimentation is to try to employ the clustering module separately, then add the result to the 2D box tracker using dwBoxTracker2D_addPreClustered(). But before doing so, it would really help to get some clarification on a few items:

  1. Can you provide any insight into the underlying cause of the two boxes depicted above being merged?
  2. The clusterer API takes a collection of input weights, one per input box. I can't find any description of their purpose in the documentation. What do these input weights do and what are reasonable defaults?
  3. The tracking API method dwBoxTracker2D_addPreClustered() takes a collection of dwTrackedBox2D instances while dwBoxTracker2D_add() simply takes a collection of dwBox2D instances. In the former case, what does the tracker do with the additional information (e.g. id and confidence) from dwTrackedBox2D?

Any help you could provide would be greatly appreciated…

Dear steve6njse,

  1. Just to confirm, In the object detect tracker sample, the yellow color box indiate the region fed into the network, did you modify the color setting.

  2. Weight is optional and corresponds to the confidence of the bounding box. If minSumOfWeights is set to higher than 0, then the sum of the confidences of bounding boxes which would form a cluster is lower than the minSumOfWeights, the cluster will be rejected.

  3. Using dwBoxTracker2D_addPreClustered will prevent the tracker from clustering the box with other overlapping boxes that are detected in the same frame.
    The id is preserved if this box is not merged with an existing tracked box, ignored otherwise.
    The confidence is multiplied by confRateDetect. If the box is not merged, then the result of this operation is the new confidence. Otherwise, the result of this operation is added to the confidence of the box that it got merged into.
    The bottom line is that this is treated like a regular box, but not clustered together with the detected boxes in the same frame

Thanks Siva for your quick response turnaround.

The yellow box in the image I provided is a box output from the DRIVE 9 tracker. That box was subsequently fed into one our internal tools for display (we use yellow to represent pedestrians).

It was produced by feeding a box output from DriveNet into the tracker using dwBoxTracker2D_add (i.e. non-clustered input). To reiterate for clarity, we are using the DriveNet detector as configured in the drivenet sample and the tracker as configured in the sample_object_detector_tracker sample.

Just to ensure we’re on the same page, we’re still looking for a potential solution to the merging problem depicted above. Or at least some avenues to pursue.

Here’s some more data. Please take a look at https://imgur.com/a/zkModuo. The following is debug data from the first frame the large depicted box is returned:

--------------------------------------------------------
--- FRAME 133
--------------------------------------------------------
[### DETECTOR ###] id=0    type=pedestrian   conf=0.563104
[### DETECTOR ###] Returning box: 48 (x=1087, y=545, w=30, h=90)
[### DETECTOR ###] id=0    type=pedestrian   conf=0.459210
[### DETECTOR ###] Returning box: 49 (x=797, y=552, w=16, h=39)
[04-03-2020 09:29:23] bounding boxes have ambiguous overlaps
[04-03-2020 09:29:23] box 4 is added to the tracker
[### TRACKER ###] id=3    type=pedestrian   conf=0.800000
[### TRACKER ###] Returning box: 3 (x=1089, y=540, w=30, h=93)
[### TRACKER ###] id=4    type=pedestrian   conf=0.900000
[### TRACKER ###] Returning box: 4 (x=797, y=545, w=320, h=90)

A (presumably false positive) pedestrian is detected with a width of 16 but once the clusterer/tracker is run, it balloons to a width of 320. Lines 8 & 9 are DW output, the rest is our output.

Dear steve6njse,
Is it possible to share the code snippet here to investigate the issue further?

Hi Siva-

I emailed some code samples to a couple of your colleagues; they should be forwarding them along to you. The two files contain a class that we have adapted from the sample_object_detector_tracker sample included along with DRIVE 9.

We instantiate one of these per object classification. So we feed pedestrian boxes output from the DriveNet detector to a pedestrian tracker, and bicycle boxes to a bicycle tracker.

I only mention this in case there is some internal state (i.e. static or singleton data maintained by the DW clusterer and/or tracker) that could be shared by both tracker instances and have the potential to cause problems.

Hi steve6njse,

to make some things clear, it is important to understand the differences between the samples you are merging:

  1. sample_object_detector_tracker
  • uses a TRT plan file to run inference on an image and output bounding boxes for all detections, meaning the output will have probably multiple bounding boxes for the same object.
  • when adding those boxes to the box tracker using dwBoxTracker2D_add the tracker expects multiple detections of the same object, and so first performs clustering to the group of boxes close-by, the average box is computed and added to be tracked
  1. sample_drivenet uses an integrated NN called DriveNet to detect objects using the api dwObjectDetector_detectObjects. the output of that api is one bounding box per object, meaning, no clustering is needed here.

So, if you wish to use DriveNet as an input for the tracker, you might want to try and iterate over all the bounding boxes detected from DriveNet and add them one by one separately by calling dwBoxTracker2D_add for each one:

for(unsigned int i = 0 ; i < m_detectedBoxList.size(); i++)
        CHECK_DW_ERROR(dwBoxTracker2D_add(m_detectedBoxList.data() + i, 1, m_boxTracker));

By that, you can prevent the tracker from merging bounding boxes that belong to different objects.

please try that and update

Siva-

Thank you for the update! Some preliminary testing shows promise! I’ll be sure and get back to you once I verify that we’re getting better results after I do some more testing.

We really appreciate the responsiveness.

-Steve

Siva-

I understand your response, but something still confuses me. At one point, I did try using:

dwBoxTracker2D_addPreClustered()

…to feed input to the tracker. Shouldn’t this have achieved an equivalent result?

Thanks,

-Steve

Hi steve6njse,

yes, this should achieve the equivalent result, but make sure you set a unique id for each dwTrackedBox2D you give the dwBoxTracker2D_addPreClustered api along the program run.
giving the tracker 2 tracked boxes in the same frame with the same ID will cause the tracker to not function correctly.

-Shay