Tracker changing final infer class ID

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) = Both
• DeepStream Version = 5.0.1
• JetPack Version (valid for Jetson only) = 4.4
• TensorRT Version = Desktop (7.0.0.11-1+cuda10.2)
• NVIDIA GPU Driver Version (valid for GPU only) = 450.66
• Issue Type( questions, new requirements, bugs) = potential bug

My setup has two engines (primary & secondary) with tracking. I’ve distilled down the issue to definitely be with the tracker. Allow me to explain.

If I have the tracker running (pgie interval=10) then on every tracked frame the object that has class 0 in the sgie will get changed into class 0 for the pgie.

Let’s say pgie class 0 is “person” and sgie class 0 is “hat”

Frame 0 (inference): hat (obj->unique_component_id = 2)
Frame 1 (track): person (obj->unique_component_id = 1)
Frame 2 (track): person (obj->unique_component_id = 1)
Frame 3 (track): person (obj->unique_component_id = 1)
Frame 4 (track): person (obj->unique_component_id = 1)
Frame 5 (track): person (obj->unique_component_id = 1)
Frame 6 (track): person (obj->unique_component_id = 1)
Frame 7 (track): person (obj->unique_component_id = 1)
Frame 8 (track): person (obj->unique_component_id = 1)
Frame 9 (track): person (obj->unique_component_id = 1)
Frame 10 (inference): hat (obj->unique_component_id = 2)
etc…

I have confirmed that this is a tracker related issue, because if I disable the tracker and set interval=0 then each and every frame correctly detects “hat” for the object. This does not happen with any other classes!

I have also tried each of the included Deepstream 5.0.1 trackers and each one exhibits the same problem in the same way.

I don’t know how to reproduce this bug, but it’s happening on both my Desktop test environment and all Jetson devices.

Any suggestions?

#3

Hey, what’s your pipeline and are you using deepstream-app or your customized app?

#4

It’s a slightly modified deepstream-app. This wasn’t occurring in DS 4.x

I can’t be sure, but I think the issue was introduced with the 5.0.1 update.

Here’s my pipeline:

NvStreamMux
NvVideoConvert
NvInfer (pgie)
NvInfer (sgie)
NvTracker
NvStreamDemux

#5

in frame#10 , I see the hat is for obj->unique_component_id = 2, that means the object is detected by sgie, so what’s the issue, I don’t really get it.

#6

that means the object is detected by sgie, so what’s the issue, I don’t really get it.

If it was detected as “hat” and stayed as a hat for all of the frames then there would be no issue.

However the issue is that the object is continually tracked across all of the intermediate frames and has the same tracking ID, but this same object oscillates between “person” and “hat” depending if it’s a tracking frame or an inference frame. On an inference frame it correctly lands on “hat” then immediately the next frame it becomes a “person” and stays as a person until the next inference frame, and the process repeats. This unexpected and undesired behaviour.

This only happens for class ID 0. The rest of the classes are both tracked correctly and maintain their own unique tracking IDs.

#7

Hey,
There are still something confued, is the tracker followed by sgie in your pipeline? I cannot get the exact issue per your description, will be better if you can share me a repro?

#8

No, the tracker happens after pgie and sgie.

Sorry, there’s no repo that can be shared.

Ok, I’ll explain with more examples (class names are arbitrary)

sgie class 1: “chair”
Frame 0 (inference): chair (obj->unique_component_id = 2)
Frame 1 (track): chair (obj->unique_component_id = 2)
Frame 2 (track): chair (obj->unique_component_id = 2)
Frame 3 (track): chair (obj->unique_component_id = 2)
Frame 4 (track): chair (obj->unique_component_id = 2)
Frame 5 (track): chair (obj->unique_component_id = 2)
Frame 6 (track): chair (obj->unique_component_id = 2)
Frame 7 (track): chair (obj->unique_component_id = 2)
Frame 8 (track): chair (obj->unique_component_id = 2)
Frame 9 (track): chair (obj->unique_component_id = 2)
Frame 10 (inference): chair (obj->unique_component_id = 2)

sgie class 2: “banana”
Frame 0 (inference): banana (obj->unique_component_id = 2)
Frame 1 (track): banana (obj->unique_component_id = 2)
Frame 2 (track): banana (obj->unique_component_id = 2)
Frame 3 (track): banana (obj->unique_component_id = 2)
Frame 4 (track): banana (obj->unique_component_id = 2)
Frame 5 (track): banana (obj->unique_component_id = 2)
Frame 6 (track): banana (obj->unique_component_id = 2)
Frame 7 (track): banana (obj->unique_component_id = 2)
Frame 8 (track): banana (obj->unique_component_id = 2)
Frame 9 (track): banana (obj->unique_component_id = 2)
Frame 10 (inference): banana (obj->unique_component_id = 2)

sgie class 3: “car”
Frame 0 (inference): car (obj->unique_component_id = 2)
Frame 1 (track): car (obj->unique_component_id = 2)
Frame 2 (track): car (obj->unique_component_id = 2)
Frame 3 (track): car (obj->unique_component_id = 2)
Frame 4 (track): car (obj->unique_component_id = 2)
Frame 5 (track): car (obj->unique_component_id = 2)
Frame 6 (track): car (obj->unique_component_id = 2)
Frame 7 (track): car (obj->unique_component_id = 2)
Frame 8 (track): car (obj->unique_component_id = 2)
Frame 9 (track): car (obj->unique_component_id = 2)
Frame 10 (inference): car (obj->unique_component_id = 2)

pgie class 0: “person”
sgie class 0: "hat
Frame 0 (inference): hat (obj->unique_component_id = 2)
Frame 1 (track): person (obj->unique_component_id = 1)
Frame 2 (track): person (obj->unique_component_id = 1)
Frame 3 (track): person (obj->unique_component_id = 1)
Frame 4 (track): person (obj->unique_component_id = 1)
Frame 5 (track): person (obj->unique_component_id = 1)
Frame 6 (track): person (obj->unique_component_id = 1)
Frame 7 (track): person (obj->unique_component_id = 1)
Frame 8 (track): person (obj->unique_component_id = 1)
Frame 9 (track): person (obj->unique_component_id = 1)
Frame 10 (inference): hat (obj->unique_component_id = 2)

Somehow, somewhere, for some reason a tracked object of sgie class 0 has it’s unique_component_id flipped from from a 2 to a 1. This only happens if the tracker is involved. If the tracker is taken out then it never happens.

#9

If you are using deepstream-app, then the pipeline should be pgie->tracker->sgie, so you had changed the deepstream-app source code, right?

#10

Yes, it is modified.

The problem I found with the tracker after pgie and before sgie is that the sgie detections are the ones I care about (and thus need to be tracked).

However, as I mentioned before this was not a problem in DS 4.x. And I haven’t verified it, but I also believe it wasn’t an issue in DS 5.0.0.

I’ve been using the same pipeline since DS 4.0

#11

@bcao any suggestions?

#12

I’m not sure whether you modified the nvinfer preprocess logic such as gst_nvinfer_process_objects(), it’s hard to say which part caused the behavior, also I think maybe you can repro it using https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/tree/master/back-to-back-detectors and share the repro with us, so we can help to look into it.

#13

@bcao yes, I know it’s a bit impossible to trouble shoot without direct access to code.

I think reproducing with the back-to-back-detectors is a good idea. I’ll try that and if I manage to reproduce the issue there I’ll share that version of the code. Thank you.