Gating Metadata Flow with probe

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.0.1
• TensorRT Version
8.0.1.6
• NVIDIA GPU Driver Version (valid for GPU only)
470.86
• Issue Type( questions, new requirements, bugs)
Question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

My problem is tightly related with Gating Metadata Flow. The only thing that is different is that I am using a probe instead of a custom plugin.

Let’s say you have a tee that’s split the pipe towards their own respective sinks; however, let’s further assume that we want one branch to continue on as normal to its sink, and the other one (just after being split) to gate in a stateful way the flow of metadata to its own sink, in order to modify the coordinates of its metadata objects.

I have tried the following without success:

static GstPadProbeReturn
msgconv_sink_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
                              gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *)info->data;
  // Create a copy of the metadata. This line does not make any difference. The metadata
  // is still modified for the two branches of the pipeline.
  if (!gst_buffer_copy_into (buf, buf, GST_BUFFER_COPY_META , 0, -1))
    g_print ("Error: Buffer metadata copy failed \n");
  
  NvDsMetaList *l_frame = NULL;
  NvDsMetaList *l_obj = NULL;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);

  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
    for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
         l_obj = l_obj->next)
    {
      NvDsObjectMeta *obj_meta = (NvDsObjectMeta *)l_obj->data;
      obj_meta->rect_params.left -= 100;
      obj_meta->rect_params.top -= 100;
    }
  }
  return GST_PAD_PROBE_OK;
}

As soon as I modify the metadata in one of the branches it gets modified also in the other. Could you please help me?

Hi @eduardo.cuesta , tee plugin is not really copy the data to every branch. It just adds the gstbuf’s refcount. So if you don’t deep copy the gstbuf, all the brach actually use the same data. If you want to realize your functions, you can try to use the gst_buffer_copy_into function to deep copy your gstbuffer.

Hi @yuweiw , I have tried the deep copy (GST_BUFFER_COPY_DEEP) and the metadata is still modified at the other pipe branch:

static GstPadProbeReturn
msgconv_sink_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
                              gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *)info->data;
  // Create a copy of the metadata. This line does not make any difference. The metadata
  // is still modified for the two branches of the pipeline.
  if (!gst_buffer_copy_into (buf, buf, GST_BUFFER_COPY_DEEP , 0, -1))
    g_print ("Error: Buffer metadata copy failed \n");
  
  ...
  return GST_PAD_PROBE_OK;
}

Any ideas?

Oh I think I am not copying the data as the function gst_buffer_copy_into does nothing if both input buffers are the same: gst_buffer_copy_into (buf, buf, GST_BUFFER_COPY_DEEP , 0, -1).

What can I do inside my probe?

From your code, you shouldn’t just copy the buf to the same buf. It makes no sense.
You can try the following method:

1.create a new gstbuffer
2.deep copy the old buffer to the new buffer
3.process your data with the new gstbuffer

I got Segmentation Fault as soon as I try to iterate over the metadata

This is working:

static GstPadProbeReturn
msgconv_sink_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
                              gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *)info->data;
  // Create a deep copy of the buffer.
  GstBuffer *new_buf = gst_buffer_copy_deep(buf);
  NvDsMetaList *l_frame = NULL;
  NvDsMetaList *l_obj = NULL;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(new_buf);

  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
    for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
         l_obj = l_obj->next)
    {
      NvDsObjectMeta *obj_meta = (NvDsObjectMeta *)l_obj->data;
      obj_meta->rect_params.left -= 100;
      obj_meta->rect_params.top -= 100;
    }
  }

  info->data = new_buf;
  gst_buffer_unref(buf);
  return GST_PAD_PROBE_OK;
}

I still have to double check the results. Thank you @yuweiw

EDIT: it works! many thanks @yuweiw!

1 Like

Finally, I have realized that this is not fully working… I am losing some metadata for the secondary classifiers. It is weird, because I have the metadata of the primary detectors but I do not have all the metadata for the secondary classifiers in latter plugins of the pipeline when I copy the buffer using the provided code. Concretely, I am lossing the classifier_type property from the classifier_meta elements of the classifier_meta_list of each object.

Basically, you only need to copy the buffer once after the tee plugin.
Could you provide your code if it’s simple to run and your pipeline diagram? Thanks

Sorry for my late response @yuweiw, I am checking this internally. Let me see what we can do. Thank you.

Hi again @yuweiw, I can share you privately a self contained docker image with the code and required artifacts, is that fine to you?

Yes. You can commit your diff with our demo code. It’s better to have a particular README file. But if there are too many changes, it may take a long time to analyze your code.

Perfect, I have share you privately the required data. Let me know if you need anything else for me. Thank you in advance.

Could you try the 2 method below? Theoretically, both are feasible.

  1. gst_buffer_copy_into (buf, buf, GST_BUFFER_COPY_META, 0, -1), set the GST_BUFFER_COPY_DEEP to GST_BUFFER_COPY_META
  2. You can try to add queue plugin after tee plugin, and after the queue plugin, you can add a nvvideoconvert plugin like below
       ------queue-------nvvideoconvert
tee
      -------queue-------

Nvvideoconvert can help you to copy buffer only when you set a different resolution.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.