GazeNet - Python Implimentation

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) AGX Orin
• DeepStream Version 6.1
• JetPack Version (valid for Jetson only) 5.01
• TensorRT Version 8.4
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) Question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Good day everyone,

I’m iterating towards a Python GazeNet solution. I have a pipe line that has a pgie, sgie and the gaze inference (nvdsvideotemplate) components. The pgie converts the bbox to a square format.

The pgie and sgie are working together and I have 3 outputs from the sgie - conv_keypoints_m80, softargmax (containing the facial landmark coords) and softargmax:1 containing the confidence levels of each keypoint.

I recognize that the suggested place to intercept the tensor data is in the sgie probe but i have ended up putting that logic in the OSD probe:

def osd_sink_pad_buffer_probe(pad,info,u_data):
ptvsd.debug_this_thread()
frame_number=0
#Intiallizing object counter with 0.
obj_counter = {
PGIE_CLASS_ID_PERSON:0,
PGIE_CLASS_ID_FACE:0,
PGIE_CLASS_ID_HELMET:0,
PGIE_CLASS_ID_NOHELMET:0
}
num_rects=0

gst_buffer = info.get_buffer()
if not gst_buffer:
    print("Unable to get GstBuffer ")
    return

# Retrieve batch metadata from the gst_buffer
# Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
# C address of gst_buffer as input, which is obtained with hash(gst_buffer)
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
    try:
        # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
        # The casting is done by pyds.NvDsFrameMeta.cast()
        # The casting also keeps ownership of the underlying memory
        # in the C code, so the Python garbage collector will leave
        # it alone.
       frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
    except StopIteration:
        break


    frame_number=frame_meta.frame_num
    num_rects = frame_meta.num_obj_meta
    l_obj=frame_meta.obj_meta_list

    while l_obj is not None:
        try:
            # Casting l_obj.data to pyds.NvDsObjectMeta
            obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
        except StopIteration:
            break   

        ## User meta
        l_user = obj_meta.obj_user_meta_list
        while l_user is not None:
            try:
                user_meta = pyds.NvDsUserMeta.cast(l_user.data)
            except StopIteration:
                break
            if(user_meta.base_meta.meta_type != pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META):
                continue

            
            tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)

            frame_outputs = []

            # frame_output[1] -> coords of landmarks
            # frame_output[2] -> confidence of each landmark
            output_shapes = [[80,80,80],[80,2],[80]]
            for i in range(tensor_meta.num_output_layers):
                layer = pyds.get_nvds_LayerInfo(tensor_meta, i)
                # Convert NvDsInferLayerInfo buffer to numpy array
                ptr = ctypes.cast(pyds.get_ptr(layer.buffer), ctypes.POINTER(ctypes.c_float))
                v = np.ctypeslib.as_array(ptr, shape=(output_shapes[i]))
                frame_outputs.append(v)
                #v = np.ctypeslib.as_array(ptr, shape=output_shapes[i])
            
            disp_meta = pyds.nvds_acquire_display_meta_from_pool(batch_meta)

            circle_params = disp_meta.circle_params
            circle_params[0].circle_color.set(1,0,0,1)
            circle_params[0].xc = 100
            circle_params[0].yc = 100
            circle_params[0].radius = 40
            print("")

            try:
                l_user = l_user.next
            except StopIteration:
                break
        
        pyds.nvds_add_display_meta_to_frame(frame_meta, disp_meta)

Within here I am able to pick up the tensors and cast them appropriately. I have also attempted (as a very first step) draw a simple circle on the osd (the intent here is that I will iterate over the landmarks and add them to the display meta).

My problem is that the simple circle is not drawing - I believe I have all the nvds acquire and add functions correct. I also have some additional areas of the OSD that acts on obj_meta and that is all working perfectly.

Therefore:

  1. Is it ok to draw circles (or any primitive) in the osd that uses tensor meta data (I don’t see why not)
  2. Is there a specific order that you need to draw items on the osd (i.e. from user meta first then obj meta)
  3. Any other aspects that I may have wrong

In addition, I am unable to use the cvcore routines that are in the C++ apps as I am in Python. Finally, there maybe some scaling to convert the tensor coords to the display coordinate system but I can handle that later.

Thank you for any help.

===>1. Is it ok to draw circles (or any primitive) in the osd that uses tensor meta data (I don’t see why not)
Yes, there is no problem to do this.
===>2. Is there a specific order that you need to draw items on the osd (i.e. from user meta first then obj meta)
No, there is no order to draw items.
Could you just try to draw circles in any demo that we provided? If it doesn’t work, you can attach your code. Then I can debug it directly. Thanks

Hi @yuweiw , Thank you for your response - this led me to re-inspect my code and I realized I had not incremented the num_circles and that solved the problem.

Cheers

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.