Attaching custom type metadata to gstreamer buffer on src pad causing sudden crash

I am trying to attach a custom type metadata to gstreamer buffer on nvdecoder_src_pad, exactly like in example “sources/apps/sample_apps/deepstream-gst-metadata-test/deepstream_gst_metadata.c”. For this, I have slightly modified to make the code work like bellow::

typedef struct _NvDecoderMeta
{
  cv::Mat frame;
  guint frame_type;
  guint frame_num;
  gboolean dec_err;
} NvDecoderMeta;

I have modified the nvinfer and nvdecoder src_pad_buffer_probe like this::

static GstPadProbeReturn
nvinfer_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
    gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *) info->data;
  NvDsMetaList * l_frame = NULL;
  NvDsUserMeta *user_meta = NULL;
  NvDecoderMeta * decoder_meta = NULL;
  NvDsMetaList * l_user_meta = NULL;

  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);

    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
        NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

        for (l_user_meta = frame_meta->frame_user_meta_list; l_user_meta != NULL;
            l_user_meta = l_user_meta->next) {
          user_meta = (NvDsUserMeta *) (l_user_meta->data);
         // if(user_meta->base_meta.meta_type == NVDS_DECODER_GST_META_EXAMPLE)
        //  {
            decoder_meta = (NvDecoderMeta *)user_meta->user_meta_data;
            g_print("Dec Meta retrieved as NVDS USER METADTA For Frame_Num = %d  \n",
                decoder_meta->frame_num);
            g_print("frame type = %d, frame_num = %d decode_error_status = %d\n\n",
                decoder_meta->frame_type, decoder_meta->frame_num,
                decoder_meta->dec_err);

	    cv::Mat received = decoder_meta->frame;
            cv::imwrite(patH, received);
            received.release();

          //}
        }
    }
    return GST_PAD_PROBE_OK;
}

static GstPadProbeReturn
nvdecoder_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
    gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *) info->data;
  NvDsMeta *meta = NULL;

  NvDecoderMeta *decoder_meta = (NvDecoderMeta *)g_malloc0(sizeof(NvDecoderMeta));
  if(decoder_meta == NULL)
  {
    return GST_FLOW_ERROR;
  }
  /* Add dummy metadata */
  decoder_meta->frame =  cv::imread("image.jpg");  // Set custom metadata
  decoder_meta->frame_type = (frame_number % 3);
  decoder_meta->frame_num = frame_number++;
  decoder_meta->dec_err = ((frame_number % 4) / 3);

  /* Attach decoder metadata to gst buffer using gst_buffer_add_nvds_meta() */
  meta = gst_buffer_add_nvds_meta (buf, decoder_meta, NULL,
      decoder_meta_copy_func, decoder_meta_release_func);

  /* Set metadata type */ // Commented on Purpose
 // meta->meta_type = (GstNvDsMetaType)NVDS_DECODER_GST_META_EXAMPLE;

  /* Set transform function to transform decoder metadata from Gst meta to
   * nvds meta */
  meta->gst_to_nvds_meta_transform_func = decoder_gst_to_nvds_meta_transform_func;

  /* Set release function to release the transformed nvds metadata */
  meta->gst_to_nvds_meta_release_func = decoder_gst_nvds_meta_release_func;

  g_print("GST Dec Meta attached with gst decoder output buffer for Frame_Num = %d\n",
      decoder_meta->frame_num);
  g_print("frame type = %d, frame_num = %d kitty = %c decode_error_status = %d\n\n",
      decoder_meta->frame_type, decoder_meta->frame_num, decoder_meta->kitty,
      decoder_meta->dec_err);

  return GST_PAD_PROBE_OK;
}

The rest of the code remains same exactly like the example sources/apps/sample_apps/deepstream-gst-metadata-test

The code compiles and runs as expected for first few frames, and then crashes due to memory. It even receives the cv::Mat frame on nvinfer_src_pad_buffer_probe because I see it writes the frame in the directory. But I wonder why this crashes because if I define a new ‘gchar’ type metadata instead of cv:;Mat, the program runs fine. From the DS4.0 Plugin manual, I see the line “If gst meta is not attached with gst_buffer_add_nvds_meta (), it is not transformed into DeepStream metadata. It is still available in the Gst Buffer though.” But when I comment out the line

meta = gst_buffer_add_nvds_meta (buf, decoder_meta, NULL,
      decoder_meta_copy_func, decoder_meta_release_func);

,the program crashes at the beginning.

Is it not possible to define custom metadata or am I missing something?

Why you comment meta data type “NVDS_DECODER_GST_META_EXAMPLE” ??
There are other metadata type in user_meta but not only NvDecoderMeta

Hi,
The definition looks to be a data pointer:

cv::Mat frame;

And it is assigned to a dynamic memory:

decoder_meta->frame =  cv::imread("image.jpg");  // Set custom metadata

You may try to allocate static memory, decode the image to the memory, and set decoder_meta->frame to the memory. Don’t have much experience of using OpenCV APIs. Other users may share their experience on this implementation.

Okey, I have managed to solve the problem by un-commenting meta_type on buffer_probe

meta->meta_type = (GstNvDsMetaType)NVDS_DECODER_GST_META_EXAMPLE;

but now I want to attach the metadata at frame level from different component, e.g. like in deepstream_sdk_v4.0_jetson/sources/apps/sample_apps/deepstream-user-metadata-test/deepstream_user_metadata_app.c where user_meta attached on nvinfer src element and metadata received from osd sink element. I have tried by changing deepstream-gst-metadata-test.c

decoder_src_pad = gst_element_get_static_pad (pgie, "src");
       gst_pad_add_probe (decoder_src_pad, GST_PAD_PROBE_TYPE_BUFFER,
           nvdecoder_src_pad_buffer_probe, NULL, NULL);

infer_src_pad = gst_element_get_static_pad (nvosd, "sink");
    gst_pad_add_probe (infer_src_pad, GST_PAD_PROBE_TYPE_BUFFER,
           nvinfer_src_pad_buffer_probe, NULL, NULL);

But hence the code quotes-

/* nvdecoder_src_pad_buffer_probe() will attach decoder metadata to gstreamer
 * buffer on src pad. The decoder can not attach to NvDsBatchMeta metadata because
 * batch level metadata is created by nvstreammux component. The decoder
 * component is present is before nvstreammmux. So it attached the metadata
 * using gstnvdsmeta API's.

I am unable to retrieve the attached metadata on nvinfer_src_pad_buffer_probe

Could anyone suggest how to add the metadata at frame level apart from the decoder element? I have also tried deepstream_user_metadata_app.c but it has limitations like only int or char type user_meta is supported.

Hi,
We are able to get correct NvDecoderMeta in source pad of nvosd by making the one line modification:

/* Lets add probe at decoder src pad to attach dummy decoder metadata using
   * gstnvdsmeta APIs.
   * This metadata is transformed into nvdsmeta and set as user metadata at
   * frame level */
  infer_src_pad = gst_element_get_static_pad (<b>/*pgie*/nvosd</b>, "src");
  if (!infer_src_pad)
    g_print ("Unable to get source pad\n");
  else
    gst_pad_add_probe (infer_src_pad, GST_PAD_PROBE_TYPE_BUFFER,
        nvinfer_src_pad_buffer_probe, NULL, NULL);

You may put a data pointer to the structure:

typedef struct _NvDecoderMeta
{
  guint frame_type;
  guint frame_num;
  gboolean dec_err;
  <b>void *ptr;</b>
} NvDecoderMeta;

And allocate a memory to put any data. Once decoder_gst_nvds_meta_release_func() is called, the memory can be released.

Hi @DaneLLL, yes I also confirm that’s possible. But my point is, I want to attach the meta to NvDsBatchMeta metadata from pgie and retrieve it on nvosd. I am facing problem attaching the meta from pgie at batch level keeping the nvdecoder_src_pad_buffer_probe() function same.

decoder_src_pad = gst_element_get_static_pad (/*decoder*/pgie, "src");
       gst_pad_add_probe (decoder_src_pad, GST_PAD_PROBE_TYPE_BUFFER,
           nvdecoder_src_pad_buffer_probe, NULL, NULL);

infer_src_pad = gst_element_get_static_pad (/*pgie*/nvosd, "src");
    gst_pad_add_probe (infer_src_pad, GST_PAD_PROBE_TYPE_BUFFER,
           nvinfer_src_pad_buffer_probe, NULL, NULL);

Hi,
We have one more sample:

deepstream_sdk_v4.0_jetson\sources\apps\sample_apps\deepstream-user-metadata-test

And below modification for your reference:

static GstPadProbeReturn
pgie_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
    gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *) info->data;
  NvDsMetaList * l_frame = NULL;
  NvDsUserMeta *user_meta = NULL;
  NvDsMetaType user_meta_type = NVDS_DECODER_GST_META_EXAMPLE;

  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);

  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
    l_frame = l_frame->next) {
      NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

      /* Acquire NvDsUserMeta user meta from pool */
      user_meta = nvds_acquire_user_meta_from_pool(batch_meta);

      NvDecoderMeta *decoder_meta = (NvDecoderMeta *)g_malloc0(sizeof(NvDecoderMeta));
      /* Add dummy metadata */
      decoder_meta->frame_type = (frame_number % 3);
      decoder_meta->frame_num = frame_number++;
      decoder_meta->dec_err = ((frame_number % 4) / 3);

      /* Set NvDsUserMeta below */
      user_meta->user_meta_data = (void *)decoder_meta;
      user_meta->base_meta.meta_type = user_meta_type;
      user_meta->base_meta.copy_func = (NvDsMetaCopyFunc)decoder_gst_to_nvds_meta_transform_func;
      user_meta->base_meta.release_func = (NvDsMetaReleaseFunc)decoder_gst_nvds_meta_release_func;

      /* We want to add NvDsUserMeta to frame level */
      nvds_add_user_meta_to_frame(frame_meta, user_meta);
  }
  return GST_PAD_PROBE_OK;
}