How to get result of secondary-gie?

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version Deepstream 5.0

I am using deepstream-app to run a pipeline with a primary-gie and secondary-gie. Both are classifiers.

I am able to access the results in NvDsLabelInfo for the first classifer.

How can I access the results from the secondary classifier?

Here is my ds_config file:

[primary-gie]
enable=1
gie-unique-id=1
batch-size=1
config-file=secondary/inference_config.txt
labelfile-path=primary/labels.txt

[secondary-gie0]
enable=1
gie-unique-id=2
batch-size=1
operate-on-gie-id=1
operate-on-class-ids=0;1
config-file=secondary/inference_config.txt
labelfile-path=secondary/labels.txt

From README of deepstream-test2:

Each of the nvinfer elements attach some MetaData to the buffer. By attaching
the probe function at the end of the pipeline, one can extract meaningful
information from these inferences. 

My question is: how to identify and access the metadata structures that were attached by the secondary nvinfer elements? The source code of osd_sink_pad_buffer_probe is not clear on how to use the secondary metadata structures.

Does anyone have any information on this?

There is a unique id(gie-unique-id) for every nvinfer element

@bcao Thank you for your response.

I should be more specific. When reading the metadata, e.g. NvDsObjectMeta or NvDsClassificationMeta how can I differentiate between the data added by pgie and that added by sgie?

@bcao Can you point me to some documentation on how to use the SGIE element?

The nvinfer documentation does not address how to access metadata attached by SGIE element.

I have read through the source code for both deepstream-test2 and back-to-back-detector sample code. The source code does not address how to access metadata structs added by the SGIE elements.

Any documentation or other references would be greatly appreciated.

Good question, There is a unique_component_id inside NvDsObjectMeta, please check the nvdsmeta.h for more details.

/** Holds a unique component ID that identifies the metadata in this structure. */ 
gint unique_component_id

Thank you @bcao. If I understand correctly, the SGIE adds an element to NvDsObjectMetaList, and the unique_component_id is the same as the gie-unique-id as specified in the inference_config.txt file? Is this correct?

It is still not clear how to access the SGIE result. Do you both have any more specific answers?

@bcao @mattcarp88
I just found out that SGI is a classifier so that all the information is in the obj_meta.classifier_meta_list, sth like below code:

            # Color Classifier
            l_class_meta = obj_meta.classifier_meta_list
            while l_class_meta:
                class_meta = pyds.NvDsClassifierMeta.cast(l_class_meta.data)
                l_label = class_meta.label_info_list
                while l_label:
                    label_info = pyds.NvDsLabelInfo.cast(l_label.data)
                    if label_info:
                        print(label_info.result_label)
                    l_label = l_label.next
                l_class_meta = l_class_meta.next

@1733208392 Please create a new topic for your question.

I thought it is the same topic, isnt it?

I have the same problem about accessing the second/third classifier metadata.

Do you guys have a working example? @bcao @mattcarp88 @1733208392

def osd_sink_pad_buffer_probe(pad, info, u_data):
frame_number = 0
# Intiallizing object counter with 0.
obj_counter = {
PGIE_CLASS_ID_VEHICLE: 0,
PGIE_CLASS_ID_PERSON: 0,
PGIE_CLASS_ID_BICYCLE: 0,
PGIE_CLASS_ID_ROADSIGN: 0
}
is_first_object = True
gst_buffer = info.get_buffer()
if not gst_buffer:
print("Unable to get GstBuffer ")
return

# Retrieve batch metadata from the gst_buffer
# Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
# C address of gst_buffer as input, which is obtained with hash(gst_buffer)
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
if not batch_meta:
    return Gst.PadProbeReturn.OK
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
    try:
        # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
        # The casting is done by pyds.NvDsFrameMeta.cast()
        # The casting also keeps ownership of the underlying memory
        # in the C code, so the Python garbage collector will leave
        # it alone.
        frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
    except StopIteration:
        continue

    '''
    print("Frame Number is ", frame_meta.frame_num)
    print("Source id is ", frame_meta.source_id)
    print("Batch id is ", frame_meta.batch_id)
    print("Source Frame Width ", frame_meta.source_frame_width)
    print("Source Frame Height ", frame_meta.source_frame_height)
    print("Num object meta ", frame_meta.num_obj_meta)
    '''
    frame_number = frame_meta.frame_num
    l_obj = frame_meta.obj_meta_list
    while l_obj is not None:
        try:
            obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
        except StopIteration:
            continue

        # Update the object text display
        txt_params = obj_meta.text_params

        # Set display_text. Any existing display_text string will be
        # freed by the bindings module.

        obj_counter[obj_meta.class_id] += 1

        # Font , font-color and font-size
        txt_params.font_params.font_name = "Serif"
        txt_params.font_params.font_size = 10
        # set(red, green, blue, alpha); set to White
        txt_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0);

        # Text background color
        txt_params.set_bg_clr = 1

        color_str = ""
        type_str = ""
        # Color and type Classifier, this only works for the vehicle
        if obj_meta.class_id == PGIE_CLASS_ID_VEHICLE:
            l_class_meta = obj_meta.classifier_meta_list
            while l_class_meta:
                class_meta = pyds.NvDsClassifierMeta.cast(l_class_meta.data)
                l_label = class_meta.label_info_list
                while l_label:
                    label_info = pyds.NvDsLabelInfo.cast(l_label.data)
                    print("Label Id: ", label_info.label_id, "Class: ", label_info.result_class_id, "result: ",
                          label_info.result_label)
                    # defined in the config file Color SGIE:
                    if label_info and class_meta.unique_component_id == 2:  ##Color
                        color_str = label_info.result_label
                    elif label_info and class_meta.unique_component_id == 4:  ##Type
                        type_str = label_info.result_label
                    l_label = l_label.next
                l_class_meta = l_class_meta.next

        display_str = ("%d:%s.%s.%s" % (long_to_int(obj_meta.object_id), pgie_classes_str[obj_meta.class_id], color_str, type_str))
        txt_params.display_text = display_str

        # Ideally NVDS_EVENT_MSG_META should be attached to buffer by the
        # component implementing detection / recognition logic.
        # Here it demonstrates how to use / attach that meta data.
        if frame_number % 30 == 0:
            # Frequency of messages to be send will be based on use case.
            # Here message is being sent for first object every 30 frames.

            # Allocating an NvDsEventMsgMeta instance and getting reference
            # to it. The underlying memory is not manged by Python so that
            # downstream plugins can access it. Otherwise the garbage collector
            # will free it when this probe exits.
            msg_meta = pyds.alloc_nvds_event_msg_meta()
            msg_meta.bbox.top = obj_meta.rect_params.top
            msg_meta.bbox.left = obj_meta.rect_params.left
            msg_meta.bbox.width = obj_meta.rect_params.width
            msg_meta.bbox.height = obj_meta.rect_params.height
            msg_meta.frameId = frame_number
            msg_meta.trackingId = long_to_int(obj_meta.object_id)
            msg_meta.confidence = obj_meta.tracker_confidence
            color_str = ""
            type_str = ""
            # Color and type Classifier, this only works for the vehicle
            if obj_meta.class_id == PGIE_CLASS_ID_VEHICLE:
                l_class_meta = obj_meta.classifier_meta_list
                while l_class_meta:
                    class_meta = pyds.NvDsClassifierMeta.cast(l_class_meta.data)
                    l_label = class_meta.label_info_list
                    while l_label:
                        label_info = pyds.NvDsLabelInfo.cast(l_label.data)
                        print("Label Id: ", label_info.label_id, "Class: ", label_info.result_class_id, "result: ",
                              label_info.result_label)
                        # defined in the config file Color SGIE:
                        if label_info and class_meta.unique_component_id == 2: ##Color
                            color_str = label_info.result_label
                        elif label_info and class_meta.unique_component_id == 4: ##Type
                            type_str = label_info.result_label
                        l_label = l_label.next
                    l_class_meta = l_class_meta.next

            msg_meta = generate_event_msg_meta(msg_meta, obj_meta.class_id, color_str, type_str)
            user_event_meta = pyds.nvds_acquire_user_meta_from_pool(batch_meta)
            if (user_event_meta):
                user_event_meta.user_meta_data = msg_meta;
                user_event_meta.base_meta.meta_type = pyds.NvDsMetaType.NVDS_EVENT_MSG_META
                # Setting callbacks in the event msg meta. The bindings layer
                # will wrap these callables in C functions. Currently only one
                # set of callbacks is supported.
                pyds.user_copyfunc(user_event_meta, meta_copy_func)
                pyds.user_releasefunc(user_event_meta, meta_free_func)
                pyds.nvds_add_user_meta_to_frame(frame_meta, user_event_meta)
            else:
                print("Error in attaching event meta to buffer\n")

            is_first_object = False
        try:
            l_obj = l_obj.next
        except StopIteration:
            break
    try:
        l_frame = l_frame.next
    except StopIteration:
        break

print("Frame Number =", frame_number, "Vehicle Count =", obj_counter[PGIE_CLASS_ID_VEHICLE], "Person Count =",
      obj_counter[PGIE_CLASS_ID_PERSON])
return Gst.PadProbeReturn.OK