Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) = Both • DeepStream Version = 5.0.1 • JetPack Version (valid for Jetson only) = 4.4 • TensorRT Version = Desktop (7.0.0.11-1+cuda10.2) • NVIDIA GPU Driver Version (valid for GPU only) = 450.66 • Issue Type( questions, new requirements, bugs) = potential bug
My setup has two engines (primary & secondary) with tracking. I’ve distilled down the issue to definitely be with the tracker. Allow me to explain.
If I have the tracker running (pgie interval=10) then on every tracked frame the object that has class 0 in the sgie will get changed into class 0 for the pgie.
Let’s say pgie class 0 is “person” and sgie class 0 is “hat”
Frame 0 (inference): hat (obj->unique_component_id = 2)
Frame 1 (track): person (obj->unique_component_id = 1)
Frame 2 (track): person (obj->unique_component_id = 1)
Frame 3 (track): person (obj->unique_component_id = 1)
Frame 4 (track): person (obj->unique_component_id = 1)
Frame 5 (track): person (obj->unique_component_id = 1)
Frame 6 (track): person (obj->unique_component_id = 1)
Frame 7 (track): person (obj->unique_component_id = 1)
Frame 8 (track): person (obj->unique_component_id = 1)
Frame 9 (track): person (obj->unique_component_id = 1)
Frame 10 (inference): hat (obj->unique_component_id = 2)
etc…
I have confirmed that this is a tracker related issue, because if I disable the tracker and set interval=0 then each and every frame correctly detects “hat” for the object. This does not happen with any other classes!
I have also tried each of the included Deepstream 5.0.1 trackers and each one exhibits the same problem in the same way.
I don’t know how to reproduce this bug, but it’s happening on both my Desktop test environment and all Jetson devices.
in frame#10 , I see the hat is for obj->unique_component_id = 2, that means the object is detected by sgie, so what’s the issue, I don’t really get it.
that means the object is detected by sgie, so what’s the issue, I don’t really get it.
If it was detected as “hat” and stayed as a hat for all of the frames then there would be no issue.
However the issue is that the object is continually tracked across all of the intermediate frames and has the same tracking ID, but this same object oscillates between “person” and “hat” depending if it’s a tracking frame or an inference frame. On an inference frame it correctly lands on “hat” then immediately the next frame it becomes a “person” and stays as a person until the next inference frame, and the process repeats. This unexpected and undesired behaviour.
This only happens for class ID 0. The rest of the classes are both tracked correctly and maintain their own unique tracking IDs.
Hey,
There are still something confued, is the tracker followed by sgie in your pipeline? I cannot get the exact issue per your description, will be better if you can share me a repro?
sgie class 3: “car”
Frame 0 (inference): car (obj->unique_component_id = 2)
Frame 1 (track): car (obj->unique_component_id = 2)
Frame 2 (track): car (obj->unique_component_id = 2)
Frame 3 (track): car (obj->unique_component_id = 2)
Frame 4 (track): car (obj->unique_component_id = 2)
Frame 5 (track): car (obj->unique_component_id = 2)
Frame 6 (track): car (obj->unique_component_id = 2)
Frame 7 (track): car (obj->unique_component_id = 2)
Frame 8 (track): car (obj->unique_component_id = 2)
Frame 9 (track): car (obj->unique_component_id = 2)
Frame 10 (inference): car (obj->unique_component_id = 2)
pgie class 0: “person”
sgie class 0: "hat
Frame 0 (inference): hat (obj->unique_component_id = 2)
Frame 1 (track): person (obj->unique_component_id = 1)
Frame 2 (track): person (obj->unique_component_id = 1)
Frame 3 (track): person (obj->unique_component_id = 1)
Frame 4 (track): person (obj->unique_component_id = 1)
Frame 5 (track): person (obj->unique_component_id = 1)
Frame 6 (track): person (obj->unique_component_id = 1)
Frame 7 (track): person (obj->unique_component_id = 1)
Frame 8 (track): person (obj->unique_component_id = 1)
Frame 9 (track): person (obj->unique_component_id = 1)
Frame 10 (inference): hat (obj->unique_component_id = 2)
Somehow, somewhere, for some reason a tracked object of sgie class 0 has it’s unique_component_id flipped from from a 2 to a 1. This only happens if the tracker is involved. If the tracker is taken out then it never happens.
@bcao yes, I know it’s a bit impossible to trouble shoot without direct access to code.
I think reproducing with the back-to-back-detectors is a good idea. I’ll try that and if I manage to reproduce the issue there I’ll share that version of the code. Thank you.
I came across the same issue.
nvinfer(vehicle, class=0) (unique_component_id=1) -----> nvinfer(person class=0)(unique_component_id=2) ------> nvtrack -----> nvinfer(vehicle color) ------> nvinfer(vehicle type)
finally, I got memory leak and sometimes segment fault。
when segment fault happend, the person object_meta’s unique_componet_id was 1
Just to update this. I still have this issue. I never managed to resolve it, but I did manage to create a workaround, then I moved on.
The workaround is to keep tabs on the track ID for each detection that has the correct class on the inference frame. Then on each subsequent tracking frame you check if the same class ID has had its unique_component_id changed and then manually correct it. This works because the track ID for each object will be unique across all inference engines.
Here’s some sample/pseudo code:
// my use case has far less than 64 instances of class 0 ever appearing at
// once, change this to whatever you need
#define MAX_BADCLASS_FIX 64
// the 12 in the first array dimension is for source ID, my use case
// uses 12 or fewer streams, change this to the max number of
// sources each instance of Deepstream will have
static guint badClassFix[12][MAX_BADCLASS_FIX];
guint bc_fix_counter = 0;
static void all_bbox_generated(AppCtx *appCtx, GstBuffer *buf, NvDsBatchMeta *batch_meta, guint index) {
//........... other code
// "name" and "class_id" are used later to send the results
// out, change these according to your needs
std::string name;
int class_id = 0;
if (obj->unique_component_id == 1) {
switch (obj->class_id) {
case 0:
for (int i = 0; i < MAX_BADCLASS_FIX; i++) {
if (badClassFix[frame_meta->source_id][i] == obj->object_id) {
obj->unique_component_id = 2;
name = "GOOD CLASS";
strcpy(obj->text_params.display_text, "Good Class");
class_id = CLASS_GOOD; // just an static const int reference for my classes
break;
}
}
break;
case 2:
etc....
}
}
else if (obj->unique_component_id == 2) {
switch (obj->class_id) {
case 0:
badClassFix[frame_meta->source_id][bc_fix_counter] = obj->object_id;
if (bc_fix_counter == MAX_BADCLASS_FIX) {
bc_fix_counter = 0;
}
else {
bc_fix_counter++;
}
break;
case 1:
etc....
}
}
}
This logic could definitely be optimized more, but it’s a quick and dirty solution and works perfectly in my use case.
// my use case has far less than 64 instances of class 0 ever appearing at
// once, change this to whatever you need
#define MAX_BADCLASS_FIX 64
// the 12 in the first array dimension is for source ID, my use case
// uses 12 or fewer streams, change this to the max number of
// sources each instance of Deepstream will have
static guint badClassFix[12][MAX_BADCLASS_FIX];
guint bc_fix_counter = 0;
static void all_bbox_generated(AppCtx *appCtx, GstBuffer *buf, NvDsBatchMeta *batch_meta, guint index) {
//........... other code
// "name" and "class_id" are used later to send the results
// out, change these according to your needs
std::string name;
int class_id = 0;
if (obj->unique_component_id == 1) {
switch (obj->class_id) {
case 0:
for (int i = 0; i < MAX_BADCLASS_FIX; i++) {
if (badClassFix[frame_meta->source_id][i] == obj->object_id) {
obj->unique_component_id = 2;
name = "GOOD CLASS";
strcpy(obj->text_params.display_text, "Good Class");
class_id = CLASS_GOOD; // just an static const int reference for my classes
break;
}
}
break;
case 2:
etc....
}
}
else if (obj->unique_component_id == 2) {
switch (obj->class_id) {
case 0:
badClassFix[frame_meta->source_id][bc_fix_counter] = obj->object_id;
if (bc_fix_counter == MAX_BADCLASS_FIX) {
bc_fix_counter = 0;
}
else {
bc_fix_counter++;
}
break;
case 1:
etc....
}
}
}
Acording to your workaround, it seems that the issue would happen only on the unique_component_id(2).
Is it to be understanded that first foreach unique_component_id(2)'s objects to find out the bad classes then foreach unique_component_id(1) to fix bad class?
Take a look at the previous posts for a detailed understanding on the issue. But I’ll summarize here. On the inference frame, the objects are detected and tagged correctly. On the following frames (tracking) before the next inference frame, the “unique_component_id” for class 0 on sgie (the second inference engine, ie unique_component_id=2) gets changed to 1.
My solution gets the tracking ID for the object and stores it. If on the next frame an object with the same tracking ID is detected with a unique_component_id of 1, then it manually changes it back to a “2”.