Filtering display artefacts (bounding boxes, annotations) in a Gstreamer/Deepstream 'osdsinkprobe'

[ganinduN]

Note: I’m using deepstream 5 due to a dependency in custom hardware

Hi,
I have a deepstream pipeline working on a live feed (it is working fine) I’m using an ‘osdsinkprobe’ pad to extract metadata.
At the display time I want to limit/change what goes into to the display. (e.g. Let’s say I want to filter out a subset of classes from a multi class classifier)

looking at the (metadata docs)[MetaData in the DeepStream SDK — DeepStream 6.1.1 Release documentation] I can see several functions that may possibly achieve this. for example (pyds_remove_object_meta_from_frame)[gst_element_send_nvevent_new_stream_reset — Deepstream Deepstream Version: 6.1.1 documentation]

These functions do not seem to remove the bounding boxes, I tried, pyds_remove_object_meta_from_frame() ← nothing happens, free_buffer() ← does not work, nvds_clear_obj_meta_list() ← clears all bboxes

Also I noticed that the
pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta) seem to have no effect (even my bbox color change seem to have somehow propagated to the C side doing the display overview, or again this might be a tell for my lack of understanding on how this works.

please help…

further info: This is a osd sink pad buffer probe, is it possible I’m looking at the wrong place ?

Cheers,
Ganindu.

EDIT:
I get the feeling that it deletes something and then looses the reference to the object list. I can find the list index of the objects then if I can point nvds_remove_obj_meta_from_frame it might work but I don’t know if there is a function to access that way.

here is my test code

    print("N objects = {}".format(objects))
    l_obj = frame_meta.obj_meta_list

    obj_id = 0
    deleted = 1
    increment  = True

    while l_obj is not None:
        try:
            obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)

            if obj_meta.class_id == 0 :
                pyds.nvds_remove_obj_meta_from_frame(frame_meta, obj_meta)
                print("class = {} object deleted , total deleted = {} (0 - car, 1 - bike, 2 - person)".format(obj_meta.class_id, deleted))
                deleted += 1
                # increment = False
            
            else:
                # skip_increment = False   <- This causes a segfault, (Try to increment only if not deleted)
                pass
                


            print(" class ID = {}, object index = {}, confidance = {}".format(obj_meta.class_id, obj_id, obj_meta.confidence))
            obj_id += 1


        except StopIteration:
            break

        try:
            if increment:
                l_obj = l_obj.next
        except StopIteration:
            break

[AastaLLL]

Hi,

Suppose you can achieve this by the filter-out-class-ids variable in the nvinfer configure file.
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#gst-nvinfer-file-configuration-specifications

Please give it a try.
Thanks.

[ganinduN]

Hi Thanks for the reply!
Making that change at the config file seems to be a solution. but what is the intended behaviour of

@ganinduN

pyds.nvds_remove_obj_meta_from_frame(frame_meta, obj_meta)

It seems to remove the first object meta from the frame and then somehow not let the list iterate over, If I understand correctly removing the object from the frame seems to cause the python object to loose the held reference, does that mean the NvDsObjectMeta.cast has to be used again and the process repeated to remove a subsequent object (if someone happened to chose this method)

Cheers,
Ganindu.

[AastaLLL]

Hi,

Based on the below document, the function is to remove the given object meta from the frame metadata.

nvds_remove_obj_meta_from_frame

Do you think this might be an issue?
If yes, could you share more information about it?

Thanks

[ganinduN]

Hi Asta,
Thanks a lot for getting back to me. Unless I’m making some obvious mistake repeatedly (in the code bit I’ve shared above ) it seems strange. What I noticed is it is deleting the bounding box of the first object that matches the criteria.

@ganinduN

obj_meta.class_id == 0 :

(here I’ve done a simple comparison to demonstrate the behaviour but I’m doing a set/list comparison but same result)

and then It seems to loose the object list handle (maybe that’s caused by the object pointer being affected), In summary basically if the object (it has to be more than one that matches the criteria in the list) I want to remove is in the middle of the list it seems to fail to do it.

from my intuition I believe if the list is if size one or if the target for deletion is at the end of the list (only one item to be deleted) it would seem as it’s doing what’s needed with the underlying cause (for my pains) still being in effect

Cheers,
Ganindu.

[ganinduN]

Hope this make it more clear (Obv the lists are coming from different frames)!

Cheers,
Ganindu.

[AastaLLL]

Hi,

Not sure if we understand your question correctly.
For a specific frame, you can get the object list from the below code:

l_obj=frame_meta.obj_meta_list

l_obj contains all the objects that are returned from the nvinfer.
So you should apply pyds.nvds_remove_obj_meta_from_frame(.) to all the objects that are included in the l_obj.

For example:

l_obj=frame_meta.obj_meta_list
while l_obj is not None:
    try:
        obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
    except StopIteration:
        break

    if obj_meta.class_id == 0 :
        pyds.nvds_remove_obj_meta_from_frame(frame_meta, obj_meta)
    ...

Thanks.

[ganinduN]

Hi this is exactly what I thought but it seems that removes only one object ( bounding box) if there are multiple objects in the frame that matches our criteria (e.g. class_id == 0)

Cheers,
Ganindu.

[AastaLLL]

Hi,

Please check the example below:
We can get all the objects that belong to class #0 deleted.

...
frame_number=frame_meta.frame_num
num_rects = frame_meta.num_obj_meta
l_obj = frame_meta.obj_meta_list

while l_obj is not None:
    try:
    # Casting l_obj.data to pyds.NvDsObjectMeta
    #obj_meta=pyds.glist_get_nvds_object_meta(l_obj.data)
        obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
    except StopIteration:
        break
    print(" class ID = {}, pointer = {}, confidance = {}".format(obj_meta.class_id, obj_meta, obj_meta.confidence))

    try:
        l_obj=l_obj.next
    except StopIteration:
        break

    if obj_meta.class_id == 0 :
        pyds.nvds_remove_obj_meta_from_frame(frame_meta, obj_meta)

print("Removing all object with class=0")
l_obj = frame_meta.obj_meta_list
while l_obj is not None:
    try:
        # Casting l_obj.data to pyds.NvDsObjectMeta
        #obj_meta=pyds.glist_get_nvds_object_meta(l_obj.data)
        obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
    except StopIteration:
        break

    print(" class ID = {}, pointer = {}, confidance = {}".format(obj_meta.class_id, obj_meta, obj_meta.confidence))
    obj_counter[obj_meta.class_id] += 1
    obj_meta.rect_params.border_color.set(0.0, 0.0, 1.0, 0.0)
    try:
        l_obj=l_obj.next
    except StopIteration:
        break
...

Output:

...
Frame Number=19 Number of Objects=12 Vehicle_count=0 Person_count=3
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f76884458>, confidance = 0.9110006093978882
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fe68>, confidance = 0.8374659419059753
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fea0>, confidance = 0.8088049292564392
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f7688ffb8>, confidance = 0.54826420545578
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fdf8>, confidance = 0.44037896394729614
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fd18>, confidance = 0.4014127254486084
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fce0>, confidance = 0.3247731328010559
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fed8>, confidance = 0.2600659430027008
 class ID = 0, pointer = <pyds.NvDsObjectMeta object at 0x7f7688ff48>, confidance = 0.2544282376766205
 class ID = 2, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fd50>, confidance = 0.8948428630828857
 class ID = 2, pointer = <pyds.NvDsObjectMeta object at 0x7f7688ff80>, confidance = 0.8110334277153015
 class ID = 2, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fd88>, confidance = 0.2596331238746643
Removing all object with class=0
 class ID = 2, pointer = <pyds.NvDsObjectMeta object at 0x7f76884538>, confidance = 0.8948428630828857
 class ID = 2, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fd88>, confidance = 0.8110334277153015
 class ID = 2, pointer = <pyds.NvDsObjectMeta object at 0x7f7688fd50>, confidance = 0.2596331238746643
Frame Number=20 Number of Objects=12 Vehicle_count=0 Person_count=3
...

Thanks.

[ganinduN]

Hi Asta!!

Fantastic work! I know understand what caused my list handle to loose it’s hold! basically you moved the object reference before the destructive action so you always had the foothold to carry out the next move and so on!!

Thanks a lot,
Ganindu.