Objects instantly lost on KLT tracker

[Bazziil]

Hi,
I noticed that when i’m using the KLT Tracker algorithm, sometimes the bbox will instantly get lost by the algorithm. Why does it behaves like this ? and how to prevent it ?

For example, on the dashcam.mp4 notice how the far right blue bounding box on the panel instantly get lost on frame 2 :

I get a lot of this case on homogeneous background, with a median filter i get better results. It’s probably linked to the method used, do you have any quick suggestion on how to prevent this before i look into details on KLT ?

[AastaLLL]

Hi,

The original box found in first frame is not good.
It just covers partial of the sign so it is quite possible to lost.

You can try to adjust the parameter of KLT tracker to see if it helps first.
https://docs.nvidia.com/vpi/group__VPI__KLTFeatureTracker.html#structVPIKLTFeatureTrackerParams

Thanks.

[Bazziil]

Hi AastaLLL,

It just covers partial of the sign so it is quite possible to lost.

I tried many bbox around this sign. As KLT is heavily based on feature extraction, I have deliberately put the bounding box on the the edge of the sign. With the idea that maybe that would help it track the sign with the important contrast white/grey, but that doesn’t change the fact that it get lost in the next frame :

Here is the bbox exclusively on the sign :

I understand each KLT parameter but at what value these thresholds are compared : nccThresholdKill, nccThresholdStop, nccThresholdUpdate ?
I have already modified these parameters to get the algo to not lost bbox and to update more often the box for example. For this example, the only bbox size on the sign that doesn’t get lost with modified parameters is when it track the background. So my main goal is to prevent the loss of the object, because i can’t deal with a tracking algorithm that reject my bounding box.

Thanks.

[AastaLLL]

Hi,

Do you know the size of your bounding box?
We do have a known limitation that the tracker is not working on the bbox > 64x64.

Thanks.

[Bazziil]

Hi AastaLLL,
Yes i do know that the limitation is 64*64.
I’m using the initial script with the initial configuration, but i added : 0 980 228 34 32 in the .txt file.
I tried to vary these parameters but got no luck.
Even if the correct bbox is 0 982 231 32 31. To prevent the loss of the bbox, the only way i have for now is that i have to loop the input bbox and vary these parameters until it doesn’t get lost.

[AastaLLL]

Hi,

Could you share the video(dashcam.mp4?) and bbox txt with us so we can check it internally?
Thanks.

[Bazziil]

Hi,
Sure : dashcam.txt (731 Bytes)

The bbox concerning the sign is on the 9th lines
Thank you.

[AastaLLL]

Hi,

Thanks for the data.

We can reproduce this issue and the bbox is dropped at frame 0.
Will share more information with you later.

Thanks.

[Bazziil]

Hi AastaLLL,
Thanks for reply, I am waiting to hear from you.
If you need any help or test please let me know.

[AastaLLL]

Hi,

Thanks for your patience.

We manually disable the drop mechanism in 06-klt_tracker.

if (bboxes[b].trackingStatus == 0)
{
    std::cout << curFrame << " -> dropped " << b << std::endl;
    // bboxes[b].trackingStatus = 1;
}

And you can find the bbox return in tracker is not good.
This is the reason that the bbox is dropped intermediately.

Frame-0:

Frame-1:

Thanks.

[Bazziil]

Hi,
That’s actually how I manage to work around the fact that the object get instantly lost :

if (bboxes[b].trackingStatus == 0)
{
    std::cout << curFrame << " -> dropped " << b << std::endl;
    // bboxes[b].trackingStatus = 1;
    // Try to modify bbox or image parameters until !bboxes[b].trackingStatus = 1;
}

This does not prevent the instant loss of objects and that’s what I was asking in this topic, just look at the final video of this example :

[AastaLLL]

Hi,

Sorry that my statement is not clear enough.

The object lost is caused by the large difference between bbox of adjacency frame.
You can find an example in the comment shared at Jun 9.

This may come from the KLT assumption, which requires all bbox have similar motion.
In the video, foreground motion (cars) seems dominates the main motion.

Thanks.