How to add metadata to plugin before Gst-nvstreammux

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• DeepStream Version: 7.X

I have a pipeline which in general looks like this:

source -> decode --(I want to put some data here using probe)-> process -> streammux -> nvinfer -(I want to read these data here)-> display

In 2022 it was not supported ( Adding custom meta before New Gst-nvstreammux ) but now it seems to be.

I found a mention in docs describing Adding metadata to the plugin before Gst-nvstreammux, but there is no meaningfull example showing how to properly do it.

example

I tried to add it in the the gst probe attached with GstPad.add_probe( Gst.PadProbeType.BUFFER, self.gst_video_probe)

    def gst_video_probe(
        self,
        pad: Gst.Pad,
        info: Gst.PadProbeInfo,
        *user_data: Any,
    ):
        """Video stream probe used for manipulating stream buffers."""
        gst_buffer = info.get_buffer()
        if not gst_buffer:
            logging.error("Unable to get GstBuffer.")
            return Gst.PadProbeReturn.OK
        try:
           # somehow initialize all of the arguments
            usermeta = pyds.NvDsUserMeta()

            nvinstance: pyds.NvDsMeta = pyds.gst_buffer_add_nvds_meta(
                gst_buffer,
                usermeta,
                None,
                pyds.nvds_batch_meta_copy_func,
                pyds.nvds_batch_meta_release_func,
            )

and I got

gst_buffer_add_nvds_meta(): incompatible function arguments. 
The following argument types are supported:
    1. (buffer: _GstBuffer, meta_data: capsule, user_data: capsule, copy_func: void* (void*, void*), release_func: void (void*, void*)) -> _NvDsMeta

Invoked with: 
<Gst.Buffer object at 0x71f5337d2500 (GstBuffer at 0x71f43c326650)>, 
 <pyds.NvDsUserMeta object at 0x743fad71d9b0>, 
None, 
<built-in method nvds_batch_meta_copy_func of PyCapsule object at 0x71f558d50870>, 
<built-in method nvds_batch_meta_release_func of PyCapsule object at 0x71f558d508a0>
> /workspace/eyeq/infer/eyeq/pipeline_probe.py(196)gst_video_probe()
-> nvinstance: pyds.NvDsMeta = pyds.gst_buffer_add_nvds_meta(
3

Just from the log you attached, the 2nd parameter is None which is incompatible function arguments.
Could you refer to our C code first to learn how to use this feature?

deepstream\sources\apps\sample_apps\deepstream-gst-metadata-test
4

SIDENOTE I made mistake during copying error, there was none in 3rd , not 2nd argument

Could you refer to our C code first to learn how to use this feature?

Yes I refered to C code from your repos and it looked like this

  GstBuffer *buf = (GstBuffer *) info->data;
  NvDsMeta *meta = NULL;

  H264parseMeta *h264parse_meta = (H264parseMeta *)g_malloc0(sizeof(H264parseMeta));
  if(h264parse_meta == NULL)
  {
    return GST_FLOW_ERROR;
  }
  /* Add dummy metadata */
  h264parse_meta->parser_frame_num = parsed_frame_number++;

  /* Attach decoder metadata to gst buffer using gst_buffer_add_nvds_meta() */
  meta = gst_buffer_add_nvds_meta (
    buf, 
    h264parse_meta, 
    NULL,
    h264parse_meta_copy_func, 
    h264parse_meta_release_func);

As you can see 3rd element is marked as NULL, so this Python code should work.

5

I also checked docs for this API NVIDIA DeepStream SDK API Reference: DeepStream Metadata Extension

And according to them 2nd and 3rd arguments should contain gpointers to structures (Null values should also be valid pointers).

6

Yes. But this is just applicable to the C/C++ language.
For python, these two parameters require capsule type. So you need to use 2capsule parameters for this API.

7

@yuweiw

Do you know what capsule is?

8

You can refer to capsule and we also used the variable of capsule type in the deepstream_imagedata-multistream_cupy.py.

We currently do not have a sample for this gst_buffer_add_nvds_meta API in python. You will need to implement it yourself at present.

9

In the docs you mention mention there are C API-s for capsule, not the Python ones. Your bindings for this API seem to be broken, because my attempts to pass pointer values (like gpointer values).

I am not the first person asking about usage of this API. Use gst_buffer_add_nvds_meta() function in python - #5 by jpes

11

@yuweiw

In this post: Attaching User Metadata to GStreamer Buffer using Python Bindings - #12 by yuweiw

You stated that:

We’ll develop it and provide a guide for this in the future version.

Could you tell me where this guide is placed?

12

Another thread with the same issue: How to pass MetaData from One pipeline node to another - #11 by yuweiw

13

Guide: CUSTOMUSERMETAGUIDE.md.

Some workarounds:
How to add custom metadata in a GStreamer native plugin and access it in Python.

14

The instruction you mentioned does not address this issue, because it shows how to add custom metadata to existing NvDsUserMeta.

FileSrc -> H264Parse -> NvV4l2Decoder -> NvStreammux -> Queue -> Queue -> FakeSink
                                                    |                     |
                                                    |                     |
                                                    -probe                -probe

I want to create NvDsUserMeta, not to add something to existing instance..

15

You can not create that metadata. We have a memory pool to manage the usage of these metadata. You can only get that from the pool by the pyds.nvds_acquire_user_meta_from_pool(batch_meta).