Tracking or X/Y Coords issue with two GIEs (back to back detectors)

[cbstryker]

• Hardware Platform = Jetson and dGPU
• DeepStream Version = 4.0.2
• TensorRT Version = 6
• NVIDIA GPU Driver Version = 440.82

I’m getting an issue with either the tracking or the post-processing of the object coords. Let me explain:

Primary inference sees the object as
Left: 100, Top: 50, Width: 200, Height: 200

This is cropped and passed to the secondary inference, which then detects another object within it’s own reference at:
Left: 50, Top: 25, Width: 100, Height: 100

The end result I get is:
Left: 50, Top: 25, Width: 100, Height: 100

The final result for the object should be:
Left: 150, Top: 75, Width: 100, Height: 100

This results in bounding boxes appearing at incorrect locations in the image and incorrect data.

Here’s the pipeline output of my application

I should note that the back-to-back detectors sample application does not experience this issue. The objects are positioned correctly in the OSD and examining the object metadata is good also.

Here is the pipeline output of the back-to-back detector:

I’m not sure where the issue could be. Anyone have any ideas?

[bcao]

Where did you add the log to print the coords?
Have you added same print in back to back detector sample?

[cbstryker]

I’ve evaluated the variables for the coords at just about every location. From inside the gst-nvinfer plugin to the all_bbox_generated method.

And yes I’ve added code to print out from the back to back detector sample, but it’s not necessary as the bbox’s are in the back to back sample (using my modified version of the gst-nvinfer as the second detector) works fine.

[bcao]

Could you check the following logic?

gstnvinfer_meta_utils.cpp->
attach_metadata_detector->
if(!nvinfer->process_full_frame) {

      rect_params.left += parent_obj_meta->rect_params.left;

      rect_params.top += parent_obj_meta->rect_params.top;

    }

[cbstryker]

I’ve checked the logic and that if statement is evaluating to “false”, but this put me on the right track.

In gstnvinfer.cpp at lines 1642 to 1645:

if (nvinfer->process_full_frame) {
      flow_ret = gst_nvinfer_process_full_frame (nvinfer, inbuf, in_surf);
  } else {
      flow_ret = gst_nvinfer_process_objects (nvinfer, inbuf, in_surf);
  }

I find this odd as in the same file we have:

#define PROCESS_MODEL_FULL_FRAME 1
#define PROCESS_MODEL_OBJECTS 2

#define DEFAULT_PROCESS_MODE PROCESS_MODEL_FULL_FRAME

At line 306:

nvinfer->process_full_frame = DEFAULT_PROCESS_MODE;

So the if statement “if (nvinfer->process_full_frame)” will always evaluate to true as both modes are a non-zero value, so it will always process in full frame mode.

If I throw in the following in front of line 1642 then it works as desired (in object only mode) as the if statement evaluates to false and calls “gst_nvinfer_process_objects”:

nvinfer->process_full_frame = 0;

However, I must be missing something since the back-to-back detector does seem to be working properly.

My custom application is based off of the deepstream-app example. So I’m also using the config parser within that sample. Maybe something isn’t being parsed correctly there?

[bcao]

Could you share your sgie config file with us?
In addition, if you want us to help check your modification, pls share the diff file with us instead of telling the number.

[cbstryker]

Primary Config:

[property]
model-engine-file=/tmp/model-b1.engine
gpu-id=0
net-scale-factor=1
model-color-format=0
custom-network-config=/tmp/yolov3-tiny.cfg
labelfile-path=/tmp/labels-80.txt
network-mode=0
num-detected-classes=80
gie-unique-id=1
is-classifier=0
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseCustomYoloV3Tiny
custom-lib-path=/tmp/lib/libnvdsinfer_custom_impl_Yolo.so

As for the line numbers, I was not sharing my modified version. I was referencing the original gst-nvinfer plugin code. Although the numbers may be off as I usually reformat the code with Jetbrains’ Clion.

Complete side note: I do this because I prefer:

void function () {
   // do something
}

instead of:

void
funtion ()
{
   // do something
}

As I find it neater and more compact. This is why the line numbers might not line up.

[bcao]

Is the config for sgie?

[cbstryker]

Sorry, the previous one was for PGIE.

Here’s the SGIE:

[property]
model-engine-file=/tmp/model-b1-secondary.engine
gpu-id=0
process-mode=2
net-scale-factor=1
model-color-format=0
custom-network-config=/tmp/yolov3-custom.cfg
labelfile-path=/tmp/labels.txt
network-mode=0
num-detected-classes=33
gie-unique-id=2
is-classifier=0
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseCustomYoloV3
custom-lib-path=/tmp/libnvdsinfer_custom_impl_Yolo.so

[bcao]

@cbstryker

process-mode=2

Then I think the nvinfer->process_full_frame should be false since you set process-mode=2, have you changed the related logic?

[cbstryker]

I have not changed that logic, but a non-zero value is going to evaluate to true.

I suggest you give the following a try:

#include <stdio.h>

int test_false = 0;
int test_true = 1;
int test_unknown = 2;

int main(int argc, char *argv[]) {
        if (test_false) {
                printf ("test_false = true\n");
        }
        else {
                printf ("test_false = false\n");
        }

        if (test_true) {
                printf ("test_true = true\n");
        }
        else {
                printf ("test_true = false\n");
        }

        if (test_unknown) {
                printf ("test_unknown = true\n");
        }
        else {
                printf ("test_unknown = false\n");
        }

        return 0;
}

You will see the following as the result:

test_false = false
test_true = true
test_unknown = true

[bcao]

@bcao

if(!nvinfer->process_full_frame) {

      rect_params.left += parent_obj_meta->rect_params.left;

      rect_params.top += parent_obj_meta->rect_params.top;

    }

I didn’t get, is above logic executed?
BTW, are you thinking the value of process-mode the same as nvinfer->process_full_frame? obviously they are not the same thing, could you check the code again, nvinfer is open sourced.

[cbstryker]

The following logic

if(!nvinfer->process_full_frame) {

      rect_params.left += parent_obj_meta->rect_params.left;

      rect_params.top += parent_obj_meta->rect_params.top;

}

does not execute. (I’m trying to be very explicit with what we are referring to, to avoid miscommunication).

BTW, are you thinking the value of process-mode the same as nvinfer->process_full_frame?

I understand they are different, but maybe where I’m confused is that I thought “process-mode” was what determines whether the SGIE operates on full frame or objects only. I actually tried looking for documentation on the config group settings for the [property] group, but I couldn’t find it. The closest thing I could find is a comment in the “secondary_detector_config.txt” from the back-to-back detector sample where it says “process-mode (Default=1 i.e. primary)”

So maybe a lot of what’s going on is some confusion with the settings. If so, can you please help clarify that for me?

Right now I have a working solution by forcing the value of “nvinfer->process_full_frame” to be 0 in my custom gst-nvinfer plugin. But naturally that’s not ideal.

[bcao]

@cbstryker

I actually tried looking for documentation on the config group settings for the [property] group, but I couldn’t find it.

Actually it’s in the DS Document, could you check https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_details.3.01.html#wwpID0E0OFB0HA

@cbstryker

I’m confused is that I thought “process-mode” was what determines whether the SGIE operates on full frame or objects only.

Correct, the process-mode determines whether nvinfer operates on full frame or not.

In addition, your SGIE should operate on objects only mode per your SGIE config(process-mode=2).