How to get object image detection by pgie and label recognition by sgie

Hello every one
It me again. This is my custom project: pgie --> sgie --> custom parse output classifier it work well. But i don’t now how to get object image dectection by pgie and mapping with label recognition by sgie to save or sent to server data

Thank everyone

Pgie: yolov3 tiny model
Sgie: onnx model
Run with: deepstream-app -c

HI,
1 how to get object image dectection by pgie

–> you can refer to deepstream-image-meta-test/deepstream_image_meta_test.c::sample osd_sink_pad_buffer_probe
for how to iterate through the user meta of type “NVDS_CROP_IMAGE_META” to find image crop meta and how to access it.

2 mapping with label recognition by sgie to save or sent to server data

–> You want the label classified by sgie to sent to server, correct? you can get the display text by _NvDsObjectMeta::text_params::display_text, which is one concated string classified by all sgies, you can refer to test4 sample for how to send message by broker, we support amqp, azure, kafka, also you can implement your own protocol broker adapter using interface, nvds_msgapi,
https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_details.html#wwpID0E0VN0HA

1 Like

Hi you,

  1. i will try
  2. my thread: 1 frame image -> pgie (object detection) --> sgie (process text recognition on object detected) --> parse output text label
    now i want to get and group that object with text label
    ex: save object image with name is text label.

for the second one, you also can refer to part 1 for saving object image.

1 Like

in deepstream-app Do have function osd_sink_pad_buffer_probe ?

test2 sample should be good, which include pgie and sgies. and it have this function.

1 Like

hey you, i want run with multi source can setup in file config so i think i source deepstream-app better.

So in source deepstream_app.c have function:

process_meta (AppCtx * appCtx, NvDsBatchMeta * batch_meta)

in this function Can i access buffer meta to crop object, and save file name with label->result_label ?
if can you can help me

Yes, you can add in here, refer to deepstream_image_meta_test.c::sample osd_sink_pad_buffer_probe for how to access and save.

1 Like

Thank you, i lest try now

Hi this is my process_meta function in source deepstream-app.c

#include "gstnvdsmeta.h"
#include "nvbufsurface.h"
#include "nvds_obj_encode.h"

static void
process_meta (AppCtx * appCtx, NvDsBatchMeta * batch_meta)
{
  // For single source always display text either with demuxer or with tiler
  if (!appCtx->config.tiled_display_config.enable ||
      appCtx->config.num_source_sub_bins == 1) {
    appCtx->show_bbox_text = 1;
  }

  for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
    NvDsFrameMeta *frame_meta = l_frame->data;
    for (NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;
        l_obj = l_obj->next) {
      NvDsObjectMeta *obj = (NvDsObjectMeta *) l_obj->data;
      ///*************************///
      char fileNameString[FILE_NAME_SIZE];
      const char *osd_string = "OSD";
      int obj_res_width = GST_ROUND_DOWN_2 ((int) obj->rect_params.width);
      int obj_res_height = GST_ROUND_DOWN_2 ((int) obj->rect_params.height);
      snprintf (fileNameString, FILE_NAME_SIZE, "/home/phongtsv/nvidia/%s_%d_%s_%d_%d.jpg",
          osd_string, frame_meta->source_id, obj->obj_label, obj_res_width, obj_res_height);

      NvDsUserMetaList *usrMetaList = obj->obj_user_meta_list;
      FILE *file;
      printf("save process: %s",obj->obj_label);
      while (usrMetaList != NULL) {
          NvDsUserMeta *usrMetaData = (NvDsUserMeta *) usrMetaList->data;
          printf("saving");
          if (usrMetaData->base_meta.meta_type == NVDS_CROP_IMAGE_META) {
            NvDsObjEncOutParams *enc_jpeg_image =
                (NvDsObjEncOutParams *) usrMetaData->user_meta_data;
            /* Write to File */
            file = fopen (fileNameString, "wb");
            fwrite (enc_jpeg_image->outBuffer, sizeof (uint8_t),
                enc_jpeg_image->outLen, file);
            fclose (file);
            printf("saved");
            usrMetaList = NULL;
          } else {
            usrMetaList = usrMetaList->next;
          }
      }
      ///*************************///
      gint class_index = obj->class_id;
      NvDsGieConfig *gie_config = NULL;
      gchar *str_ins_pos = NULL;

      if (obj->unique_component_id ==
          (gint) appCtx->config.primary_gie_config.unique_id) {
        gie_config = &appCtx->config.primary_gie_config;
      } else {
        for (gint i = 0; i < (gint) appCtx->config.num_secondary_gie_sub_bins;
            i++) {
          gie_config = &appCtx->config.secondary_gie_sub_bin_config[i];
          if (obj->unique_component_id == (gint) gie_config->unique_id) {
            break;
          }
          gie_config = NULL;
        }
      }
      g_free (obj->text_params.display_text);
      obj->text_params.display_text = NULL;

      if (gie_config != NULL) {
        if (g_hash_table_contains (gie_config->bbox_border_color_table,
                class_index + (gchar *) NULL)) {
          obj->rect_params.border_color =
              *((NvOSD_ColorParams *)
              g_hash_table_lookup (gie_config->bbox_border_color_table,
                  class_index + (gchar *) NULL));
        } else {
          obj->rect_params.border_color = gie_config->bbox_border_color;
        }
        obj->rect_params.border_width = appCtx->config.osd_config.border_width;

        if (g_hash_table_contains (gie_config->bbox_bg_color_table,
                class_index + (gchar *) NULL)) {
          obj->rect_params.has_bg_color = 1;
          obj->rect_params.bg_color =
              *((NvOSD_ColorParams *)
              g_hash_table_lookup (gie_config->bbox_bg_color_table,
                  class_index + (gchar *) NULL));
        } else {
          obj->rect_params.has_bg_color = 0;
        }
      }

      if (!appCtx->show_bbox_text)
        continue;

      obj->text_params.x_offset = obj->rect_params.left;
      obj->text_params.y_offset = obj->rect_params.top - 30;
      obj->text_params.font_params.font_color =
          appCtx->config.osd_config.text_color;
      obj->text_params.font_params.font_size =
          appCtx->config.osd_config.text_size;
      obj->text_params.font_params.font_name = appCtx->config.osd_config.font;
      if (appCtx->config.osd_config.text_has_bg) {
        obj->text_params.set_bg_clr = 1;
        obj->text_params.text_bg_clr = appCtx->config.osd_config.text_bg_color;
      }

      obj->text_params.display_text = g_malloc (128);
      obj->text_params.display_text[0] = '\0';
      str_ins_pos = obj->text_params.display_text;

      if (obj->obj_label[0] != '\0')
        sprintf (str_ins_pos, "%s", obj->obj_label);
      str_ins_pos += strlen (str_ins_pos);

      if (obj->object_id != UNTRACKED_OBJECT_ID) {
        /** object_id is a 64-bit sequential value;
         * but considering the display aesthetic,
         * trimming to lower 32-bits */
        if (appCtx->config.tracker_config.display_tracking_id) {
          guint64 const LOW_32_MASK = 0x00000000FFFFFFFF;
          sprintf (str_ins_pos, " %lu", (obj->object_id & LOW_32_MASK));
          str_ins_pos += strlen (str_ins_pos);
        }
      }

      obj->classifier_meta_list =
          g_list_sort (obj->classifier_meta_list, component_id_compare_func);
      for (NvDsMetaList * l_class = obj->classifier_meta_list; l_class != NULL;
          l_class = l_class->next) {
        NvDsClassifierMeta *cmeta = (NvDsClassifierMeta *) l_class->data;
        for (NvDsMetaList * l_label = cmeta->label_info_list; l_label != NULL;
            l_label = l_label->next) {
          NvDsLabelInfo *label = (NvDsLabelInfo *) l_label->data;
          if (label->pResult_label) {
            sprintf (str_ins_pos, " %s", label->pResult_label);
          } else if (label->result_label[0] != '\0') {
            sprintf (str_ins_pos, " %s", label->result_label);
          }
          str_ins_pos += strlen (str_ins_pos);
        }

      }
    }
  }
}

but in command *NvDsUserMetaList usrMetaList = obj->obj_user_meta_list; output of usrMetaList is NULL

thank you how to fix it

Why usrMetaList still NULL although i can print label by printf("save process: %s",obj->obj_label);@amycao

It’s looking for meta type NVDS_CROP_IMAGE_META, when matched, it will go into section:
NvDsObjEncOutParams *enc_jpeg_image =
(NvDsObjEncOutParams ) usrMetaData->user_meta_data;
/
Write to File */
file = fopen (fileNameString, “wb”);
fwrite (enc_jpeg_image->outBuffer, sizeof (uint8_t),
enc_jpeg_image->outLen, file);
fclose (file);
printf(“saved”);
usrMetaList = NULL;
please noted when find, after processed it’s set to NULL or will process next node.

1 Like

thank you so much