Adding margin to ROI when inferencing on secondary GIE in Deepstream 5

Hello dev forum,

I am looking for a way to add margin to ROI (bounding box detected by primary inference) before inferencing on secondary inference. PeopleNet gives nice face ROI but secondary age/gender models typically expect face and its surrounding to work properly.

I looked all around the nvinfer parameters and it looks like nvinfer does not support this out of box. Can anyone please suggest the best way to modify ROI size in the Deepstream pipeline.

Thank you very much.

Hi,

How about you change the object metadata rect_params values before it enters the secondary nvinfer?

Thank you Miguel. Should I write a custom plugin which takes object metadata and transforms it?

Hi,

The best way to do it is to create a custom plugin based on GstVideoFilter and perform the operation in transform frame in place. That way you can use element properties to control the padding.

There is the alternative of just probing the buffer from one of the source pads, doing the operation and pushing it to the corresponding sink pad, but I don’t think that’s a good idea. Besides giving me the feeling of a hacked design I think you would need an additional memory copy.

I can share with you how some parts of the custom plugin code would look like:

static void
gst_preprocess_apply_padding (GstPreprocess *preprocess, GstBuffer * buffer)
{
  gpointer state = NULL;
  GQuark nvdsmeta_quark = g_quark_from_static_string (NVDS_META_STRING);
  GstMeta *gst_meta;
  NvDsBatchMeta *batch_meta;
  NvDsMeta *dsmeta;
  NvDsFrameMeta *frame_meta;
  NvDsFrameMetaList *frame_meta_list;
  NvDsObjectMeta *object_meta;
  NvDsObjectMetaList *obj_meta_list;

  while ((gst_meta = gst_buffer_iterate_meta (buffer, &state))) {
    if (gst_meta_api_type_has_tag (gst_meta->info->api, nvdsmeta_quark)) {
      dsmeta = (NvDsMeta *) gst_meta;
      if (dsmeta->meta_type == NVDS_BATCH_GST_META) {
        batch_meta = (NvDsBatchMeta *) dsmeta->meta_data;
        frame_meta_list = batch_meta->frame_meta_list;
        while (frame_meta_list != NULL) {
          frame_meta = (NvDsFrameMeta *) frame_meta_list->data;
          obj_meta_list = frame_meta->obj_meta_list;
          while (obj_meta_list != NULL) {
            object_meta = (NvDsObjectMeta *) obj_meta_list->data;
            object_meta->rect_params.left -= preprocess->padding_left;
            object_meta->rect_params.top -= preprocess->padding_top;
            object_meta->rect_params.width += preprocess->padding_right;
            object_meta->rect_params.height += preprocess->padding_bottom;
            obj_meta_list = obj_meta_list->next;
          }
          frame_meta_list = frame_meta_list->next;
        }
        break;
      }
    }
  }
}

static GstFlowReturn
gst_preprocess_transform_ip (GstBaseTransform * trans, GstBuffer * buf)
{
  GstFlowReturn ret = GST_FLOW_OK;
  GstPreprocess *preprocess = GST_PREPROCESS (trans);\

  gst_preprocess_apply_padding (preprocess, buf);

  return ret;
}

You can use GstIdentity as reference. The rest of the work is just installing your properties and removing the logic you don’t need from identity.