Linking Object Meta in Back to Back Nvinfer

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) :- Jetson Xavier
• DeepStream Version :- 6.0

I have this pipeline :- pgie(car detector)->sgie1(LPD)->sgie2(LPR).
In a probe function (attached on src pad of SGIE2) I’m accessing the metadata and trying to create a JSON structure that contains, information about each car, for eg :- car_bounding_box, plate_bounding_box, LP_Number, confidence_score etc. , which are attached by different models. I’m trying to extract this metadata and put in a JSON.

my question is, how can I make sure that I’m extracting the metadata properly without mismatching between cars.

eg :- suppose there are two cars in a picture, the JSON gets created with car_bbox_info and plate_bbox_info of the car_1 but licence_plate_number of car_2.

I want to make sure that the way I’m doing it is correct.

GstPadProbeReturn sgie2_src_pad_buffer_probe(
    GstPad *pad, GstPadProbeInfo *info, gpointer u_data) {
    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
         l_frame = l_frame->next) {
        NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
        int offset = 0;
        if (!frame_meta) {
            continue;
        }
        for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
             l_obj = l_obj->next) {
            obj_meta = (NvDsObjectMeta *)(l_obj->data);
            if (!obj_meta) continue;

            if (obj_meta->unique_component_id == SECONDARY_DETECTOR_UID) {
                if (obj_meta->parent) {
                    /*obtaining the vehicle (PGIE) metadata*/
                    pleft = obj_meta->parent->detector_bbox_info.org_bbox_coords
                                .left;
                    ptop = obj_meta->parent->detector_bbox_info.org_bbox_coords
                               .top;
                    pright = obj_meta->parent->detector_bbox_info
                                 .org_bbox_coords.left +
                             obj_meta->parent->detector_bbox_info
                                 .org_bbox_coords.width;
                    pbottom = obj_meta->parent->detector_bbox_info
                                  .org_bbox_coords.top +
                              obj_meta->parent->detector_bbox_info
                                  .org_bbox_coords.height;
                    float car_conf = obj_meta->parent->confidence;
                    id = obj_meta->parent->object_id;

                    /*obtaining the LPD (SGIE1)metadata*/

                    sleft = obj_meta->detector_bbox_info.org_bbox_coords.left;
                    stop = obj_meta->detector_bbox_info.org_bbox_coords.top;
                    sright = obj_meta->detector_bbox_info.org_bbox_coords.left +
                             obj_meta->detector_bbox_info.org_bbox_coords.width;
                    sbottom =
                        obj_meta->detector_bbox_info.org_bbox_coords.top +
                        obj_meta->detector_bbox_info.org_bbox_coords.height;

                    plate_conf = obj_meta->confidence;
                    class_id = obj_meta->class_id;
                    valid_plate = is_inside(pcoords, scoords); /*checking if plate bbox is inside car bbox */
                }
            }
            /*obtaining the LPR (SGIE2) metadata*/
            for (l_class = obj_meta->classifier_meta_list; l_class != NULL;
                 l_class = l_class->next) {
                class_meta = (NvDsClassifierMeta *)(l_class->data);
                if (!class_meta) continue;
                if (class_meta->unique_component_id ==
                    SECONDARY_CLASSIFIER_UID) {
                    for (label_i = 0, l_label = class_meta->label_info_list;
                         label_i < class_meta->num_labels && l_label;
                         label_i++, l_label = l_label->next) {
                        label_info = (NvDsLabelInfo *)(l_label->data);
                        if (label_info) {
                            if (label_info->label_id == 0 &&
                                label_info->result_class_id == 1) {
                                g_print("Plate License %s\n",
                                        label_info->result_label);
                                label_id = label_info->result_label;
                                number = label_id;
                                plate_count = plate_count + 1;
                                number_conf = label_info->result_prob;
                                label_id = label_info->result_label;
                            }
                        }
                    }
                }
            }

	    if (/*checking some conditions here, and creating the JSON*/ ) 
          {
                j[key].push_back(  ***/*key here is object ID */***
                    {
                     {"vehicle_box", PGIE_bbox_info},
                     {"object_id", id},
                     {"lpr_confidence", number_conf},
                     {"veh_number", number},
                     {"lpd_confidence", plate_conf},
                     {"num_plate_box", sgie1_bbox_info},
                     });       
            }
        }
    }
    /*writing the JSON to a file*/
    return GST_PAD_PROBE_OK;
}

so, with this approach, what I feel is LPR is not getting properly mapped with LPD and car detector metadata. however this is the same way it is done in the NVIDIA LPR app. But there they were not trying to match the metadata from different models. Is this not the correct way ? How can I improve it ?

any update ?

Sorry for the reply, but our supporting team is on holiday and will come back later on this topic.

okay

You can try to use some variables in this structure to distinguish different cars.

cars and plates can be linked since there is a direct link obj_meta->parent. how do I make sure I’m linking the LPR correctly.

The classifier_meta_list is a variable of the obj_meta structure. So you can correspond it to the structure.

It is not clear yet. I’ll frame the question differently.

suppose we have a pipeline, which has :-> pgie(detector)->sgie1(detector)->sgei2(classifier) . . .

and say a buffer with batch size 1 is travelling through this part of the pipeline.

before pgie, NvDsBatchMeta->NvDsFrameMeta contains nothing.
then as the buffer leaves the src pad of pgie, pgie adds its metadata to the frame, and NvDsBatchMeta->NvDsFrameMeta->NvDsObjMeta appears.

after that frame leaves sgie1, sgie1 will append a new NvDsObjMeta structure to the NvDsBatchMeta->NvDsFrameMeta , and these two would be linked with parent->

question:- Is this correct ?

after this frame passes through and leaves the sgie2, which is a classifier,
question:-> would this(sgie2) add another NvDsObjMeta in the NvDsBatchMeta->NvDsFrameMeta, with all fields empty except the classifier_meta_list ??? or it will update the classifier_meta_list in the NvDsObjMeta structure created by the sgie1 element???

Yes and it will update the classifier_meta_list in the NvDsObjMeta structure created by the sgie1 element. Also, you can refer to the attach API in our open source directly: sources\gst-plugins\gst-nvinfer\gstnvinfer_meta_utils.cpp

okay. Thanks.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.