Identify a numpy frame fed into the pipeline

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello, I’d like to follow up on a topic that has been closed: Match NvDsFrameMeta with their original gstreamer Buffer

I am sending multiple numpy images through a pipeline.

I use appsrc to feed numpy arrays to my pipeline:

img = cv2.imread("/opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.jpg")
img = cv2.resize(img, (1080, 1920))
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGBA)
buffer = Gst.Buffer.new_wrapped(img.tobytes())
self.source_bin_appsrc.emit("push-buffer", buffer)

The pipeline works fine, and I am able to retrieve the numpy image as well as the metadata from an object of type NvDsBatchMeta.

The problem is that once I get the NvDsBatchMeta I need to know what was the original numpy frame that this metadata refers to. In this case, I can’t simply download the image again from the pipeline.

The reason is the following: I am implementing a REST API. Suppose I have two users, user_A and user_B. They both send their images image_A and image_B to my API. My API sends the image to the pipeline. At the end of the pipeline I get the metadata for each image. However, how can I know what are the metadata that I need to send back to user_A, and what are the metadata for user_B?

I need to store some value into the image. Example: if I could attach to image_A the value user=user_A, then, at the end of the pipeline, when I get the metadata, I could do something like metadata.user and get the user that needs to receive the metadata.

In the post I linked above, @yuweiw said that: “You can consider using NvDsUserMeta to add your own tag for each gstbuffer through the probe function.”. However, how can I create a NvDsFrameMeta with attached NvDsUserMeta from a numpy array and send it to the pipeline?

Thanks

3

You can refer our python source code:

 user_event_meta = pyds.nvds_acquire_user_meta_from_pool

https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/apps/deepstream-test4/deepstream_test_4.py

4

Hey @yuweiw , thank you for your reply.
I think that code would be useful to add metadata.
However, in my case, I have a numpy array. And before adding metadata, I need to create a NvDsFrameMeta from that numpy array. How can I do that?
Thank you

5

Ok, I got what you mean. This binding demo is in our roadmap. Your needs are similar to this：
https://forums.developer.nvidia.com/t/attaching-user-metadata-to-gstreamer-buffer-using-python-bindings/223486
We are working on it. Thanks

6

Hi @yuweiw ,

You are correct. My needs are similar to the ones mentioned in the link.
It’s great to know you are working on it. Well done!

Thank you

8

Hi @yuweiw is there any update on this?
I found an example to attach NvDsUserMeta to aNvDsBatchMeta object, but in my case I am looking to attach custom metadata to a gstreamer buffer before sending it to nvstreammux, exactly as it is done in the example /opt/nvidia/deepstream/deepstream-6.2/sources/apps/sample_apps/deepstream-gst-metadata-test/deepstream_gst_metadata.c.
However, my problem is that I am developing in Python and I don’t know how to translate the example from C to Python. Is that even possible?
I am willing to create an example to be published on the deepstream_python_apps repository if you guys can guide me on how to translate that example.

Let’s assume that I have a buffer:

numpy_image = np.random.randint((720, 1280, 3))
gst_buffer = ndarray_to_gst_buffer(numpy_image)

How can I then attach data using gst_buffer_add_nvds_meta?
Something like this is giving me errors, and I don’t know how to properly allocate the user_data:

user_data = pyds.alloc_char_buffer(36)  # how to create custom structs?
nvds_meta = pyds.gst_buffer_add_nvds_meta(  # this is raising an error
    buffer=gst_buffer,
    meta_data=nvds_meta,
    user_data=user_data,  # how can I allocate this properly?
    copy_func=copy_func,
    release_func=release_func
)

Error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-9bee79379ade>", line 1, in <module>
    runfile('/src/src/api/utils/temp.py', wdir='/src/')
  File "/root/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/root/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/src/src/api/utils/temp.py", line 42, in <module>
    nvds_meta = pyds.gst_buffer_add_nvds_meta(  # this is raising an error
TypeError: gst_buffer_add_nvds_meta(): incompatible function arguments. The following argument types are supported:
    1. (buffer: _GstBuffer, meta_data: capsule, user_data: capsule, copy_func: void* (void*, void*), release_func: void (void*, void*)) -> _NvDsMeta
Invoked with: kwargs: buffer=<Gst.Buffer object at 0x7fc0f96bd5e0 (GstBuffer at 0x3964000)>, meta_data=<pyds.NvDsMeta object at 0x7fc1ac0c1230>, user_data=60210704, copy_func=<function copy_func at 0x7fc0f96c1430>, release_func=<function release_func at 0x7fc250a18c10>

Thank you

9

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Yes. About the binding of copy_func and release_func, we have released it in the latest version. About the user_data, it’s a gpointer in C/C++ code. You can refer to the FAQ 31.2 and bind that with that patch.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.