How to match outputs of the same object after nvdsmetamux?

hyperlight · November 15, 2022, 3:53am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): GeForce RXT 3090
• DeepStream Version: 6.1.1
• TensorRT Version: 8.4.1.5
• NVIDIA GPU Driver Version (valid for GPU only): 515.65.01
• Issue Type( questions, new requirements, bugs): question

I follow the suggestion from this answer to run multiple classifiers in parallel after the detector. My pipeline has 2 sources and its high-level view is as follow:

sources -> muxer -> detector -> dexmuxer |-> muxer -> classifier-1 |                  
                                         |->         ....          | -> metamuxer -> app_sink
                                         |-> muxer -> classifier-7 |

There are 2 sources
There are 7 classifiers running in parallel
Each classifier branch uses both sources
The pipeline doesn’t have a tracker

I attached this callback at app_sink:

GstFlowReturn new_sample_dummy(GstElement *sink, gpointer *data)
{
    GstSample *sample;
    GstBuffer *buf = NULL;
    NvDsFrameMeta *frame_meta = NULL;
    NvDsObjectMeta *obj_meta = NULL;
    NvDsMetaList *l_frame = NULL;
    NvDsMetaList *l_obj = NULL;
    NvDsClassifierMetaList *l_classifier = NULL;
    NvDsClassifierMeta *class_meta = NULL;
    NvDsLabelInfoList *l_label = NULL;
    NvDsLabelInfo *label_info = NULL;

    sample = gst_app_sink_pull_sample(GST_APP_SINK(sink));
    if (gst_app_sink_is_eos(GST_APP_SINK(sink)))
    {
        g_print("EOS received in Appsink********\n");
    }

    static guint batch_id = 0;
    g_message("==============BATCH[%d]==============", batch_id);
    if (sample)
    {
        /* Obtain GstBuffer from sample and then extract metadata from it. */
        buf = gst_sample_get_buffer(sample);
        NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);

        g_message("num frames = %d", g_list_length(batch_meta->frame_meta_list));

        for (l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next)
        {
            frame_meta = (NvDsFrameMeta *)(l_frame->data);
            g_message("frame[%d] num objs = %d", frame_meta->batch_id, g_list_length(frame_meta->obj_meta_list));

            int obj_id = 0;
            for (l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next)
            {
                obj_meta = (NvDsObjectMeta *)(l_obj->data);

                if (obj_meta->class_id == PGIE_CLASS_ID_BUD)
                {
                    for (l_classifier = obj_meta->classifier_meta_list; l_classifier != NULL; l_classifier = l_classifier->next)
                    {
                        class_meta = (NvDsClassifierMeta *)(l_classifier->data);

                        for (l_label = class_meta->label_info_list; l_label != NULL; l_label = l_label->next)
                        {
                            label_info = (NvDsLabelInfo *)(l_label->data);
                            g_message("frame[%d] obj[%d] gie-id[%d] label_id[%d]", 
                            frame_meta->batch_id, obj_id, class_meta->unique_component_id, label_info->label_id);
                        }
                    }
                }

                obj_id++;
            }
        }

        batch_id++;
        gst_sample_unref(sample);
        return GST_FLOW_OK; 
    }

    return GST_FLOW_ERROR;
}

Sample output:

** Message: 19:15:56.993: ==============BATCH[363]==============
** Message: 19:15:56.993: num frames = 2
** Message: 19:15:56.993: frame[0] num objs = 14
** Message: 19:15:56.993: frame[0] obj[0] gie-id[9] label_id[0]
** Message: 19:15:56.993: frame[0] obj[1] gie-id[9] label_id[0]
** Message: 19:15:56.993: frame[0] obj[2] gie-id[7] label_id[0]
** Message: 19:15:56.993: frame[0] obj[3] gie-id[7] label_id[0]
** Message: 19:15:56.993: frame[0] obj[4] gie-id[6] label_id[0]
** Message: 19:15:56.993: frame[0] obj[5] gie-id[6] label_id[0]
** Message: 19:15:56.993: frame[0] obj[6] gie-id[5] label_id[0]
** Message: 19:15:56.993: frame[0] obj[7] gie-id[5] label_id[0]
** Message: 19:15:56.993: frame[0] obj[8] gie-id[4] label_id[0]
** Message: 19:15:56.993: frame[0] obj[9] gie-id[4] label_id[0]
** Message: 19:15:56.993: frame[0] obj[10] gie-id[3] label_id[0]
** Message: 19:15:56.993: frame[0] obj[11] gie-id[3] label_id[0]
** Message: 19:15:56.993: frame[0] obj[12] gie-id[2] label_id[0]
** Message: 19:15:56.993: frame[0] obj[13] gie-id[2] label_id[0]
** Message: 19:15:56.993: frame[1] num objs = 7
** Message: 19:15:56.994: frame[1] obj[0] gie-id[9] label_id[0]
** Message: 19:15:56.994: frame[1] obj[1] gie-id[7] label_id[0]
** Message: 19:15:56.994: frame[1] obj[2] gie-id[6] label_id[0]
** Message: 19:15:56.994: frame[1] obj[3] gie-id[5] label_id[0]
** Message: 19:15:56.994: frame[1] obj[4] gie-id[4] label_id[0]
** Message: 19:15:56.994: frame[1] obj[5] gie-id[3] label_id[0]
** Message: 19:15:56.994: frame[1] obj[6] gie-id[2] label_id[0]

From this output, I can infer that:

there are frames 2 in the batch (as expected since there are 2 sources)
frame 0 has 2 detected objects (14 / 7 = 2)
frame 1 has 1 detected object (7 /7 = 1)

However, in frame 0, there is no way for me to find out for the same detected object, what are the classifier outputs associate to it. Say, with this outputs:

frame[0] obj[0] gie-id[9] label_id[2]
frame[0] obj[1] gie-id[9] label_id[1]
frame[0] obj[2] gie-id[7] label_id[0]
frame[0] obj[3] gie-id[7] label_id[3]

I want to know the output of 2 sgies whose unique id is 9 and 7 for each detected object, say the expected output is:

detected-object1: 2, 3 (obj[0] & obj[3])
detected-object2: 1, 0 (obj[1] & obj[2])

how to I do that?

kesong · November 15, 2022, 9:03am

Can you show the configure file of nvdsmetamux?

hyperlight · November 15, 2022, 5:19pm

Hi @kesong,

Please see the find the config file bellow:

config_metamux0.txt (1.7 KB)

hyperlight · November 17, 2022, 5:54pm

Is there any update yet?

kesong · November 22, 2022, 9:51am

nvdsmetamux only mux meta data from all branches. What is the question here?

hyperlight · November 22, 2022, 7:45pm

Thank you for the reply.

So when I run multiple classifiers sequentially, it’s straightforward to know the results from all those classifiers for the same object. However, when I run multiple classifiers in parallel, I was trying to figure out how to get the same information. I thought I can use the new metamuxer plugin but it didn’t work out so I asked for guidance. I described the use case in my original question and want to know if it’s currently supported. Do I need to change the config or my use case is not supported by metamuxer?

kesong · November 23, 2022, 9:49am

Can you have a try with “obj_meta.parent”?

hyperlight · November 25, 2022, 9:08pm

Thank you for your suggestion. I tried using obj_meta.parent but they’re NULL. Is there anything else I should try?

kesong · November 28, 2022, 9:34am

Can unique_component_id in the meta data helps in you use case?

hyperlight · November 30, 2022, 7:44pm

I tried using the unique_component_id in my original post and shown them as gie-id, it doesn’t help in my use case:

** Message: 19:15:56.993: num frames = 2
** Message: 19:15:56.993: frame[0] num objs = 14
** Message: 19:15:56.993: frame[0] obj[0] gie-id[9] label_id[0]
** Message: 19:15:56.993: frame[0] obj[1] gie-id[9] label_id[0]
** Message: 19:15:56.993: frame[0] obj[2] gie-id[7] label_id[0]
** Message: 19:15:56.993: frame[0] obj[3] gie-id[7] label_id[0]
** Message: 19:15:56.993: frame[0] obj[4] gie-id[6] label_id[0]
** Message: 19:15:56.993: frame[0] obj[5] gie-id[6] label_id[0]
** Message: 19:15:56.993: frame[0] obj[6] gie-id[5] label_id[0]
** Message: 19:15:56.993: frame[0] obj[7] gie-id[5] label_id[0]
** Message: 19:15:56.993: frame[0] obj[8] gie-id[4] label_id[0]
** Message: 19:15:56.993: frame[0] obj[9] gie-id[4] label_id[0]
** Message: 19:15:56.993: frame[0] obj[10] gie-id[3] label_id[0]
** Message: 19:15:56.993: frame[0] obj[11] gie-id[3] label_id[0]
** Message: 19:15:56.993: frame[0] obj[12] gie-id[2] label_id[0]
** Message: 19:15:56.993: frame[0] obj[13] gie-id[2] label_id[0]

I figure there is no other way than using user metadata to mark the detected objects before they’re being duplicated going into the parallel branches to help with post-processing after the metamux plugin?

kesong · December 5, 2022, 8:26am

Yes, user meta should help here.

system · December 19, 2022, 8:26am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Does DeepStream support running multiple classifiers in parallel after a detector? DeepStream SDK	9	909	May 10, 2022
Metamux showed odd outputs DeepStream SDK	6	1187	December 6, 2022
Print details of classifier_meta_list meta? DeepStream SDK deepstream	8	35	September 27, 2024
DeepSORT tracker not working on multiple streams (erases detections in some frames) DeepStream SDK	6	445	December 20, 2022
Issues using the nvdemux from custom gstreamer element DeepStream SDK camera , cuda , jetson-inference , gstreamer	5	1281	October 9, 2021
DeepStream 7.0 source_id and camera-id and pad-id problems DeepStream SDK deepstream	19	90	December 14, 2024
Issues using the nvdemux from custom gstreamer element General Topics and Other SDKs camera , cuda , jetson-inference , gstreamer	1	471	August 4, 2021
Deepstream can't sink more than 2 video streams at the same time DeepStream SDK	8	1114	October 12, 2021
Problems linking multiple demuxers to a Tee'd batched stream on dGPU DeepStream SDK	17	715	September 6, 2023
Does version 6.1.1 support single pgie, parallel sgies? DeepStream SDK	4	841	November 8, 2022

How to match outputs of the same object after nvdsmetamux?

Related topics