Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) GPU • DeepStream Version 6.1.1 • JetPack Version (valid for Jetson only) • TensorRT Version 8.4.1-1+cuda11.6 • NVIDIA GPU Driver Version (valid for GPU only) 510.108.03 (nvidia-smi) • Issue Type( questions, new requirements, bugs) Question • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) • Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description) gst-nvmsgconv
I’m trying to attach custom message blob frame meta to convert to kafka payload.
I’m following the instructions on Custom Objects but the second paragraph way:
If using Deepstream GST buffers(NVDS_FRAME_META (NvDsFrameMeta) type and NVDS_OBJECT_META (NvDsObjectMeta) type) to generate payload instead of NvDsEventMsgMeta, You can attach a custom message blob(string) to payload using NVDS_CUSTOM_MSG_BLOB type by creating NvDsCustomMsgInfo object and specifying the fields message (custom message blob) and len (size of the custom message). The low-level payload generation library parses every frame metadata and checks for NVDS_CUSTOM_MSG_BLOB type and if present attaches it to the payload. This option is available when gst property msg2p-newapi = true (Using the new msgconv API).
I already set the property and I’m trying to generate a simple JSON string containing the frame_meta.frame_num for testing.
Here’s the relevant code that gives me an error:
# # Create C-memory-managed generic buffer through bindings
custom_msg_buffer = pyds.alloc_buffer(len(custom_json_str))
print("custom msg buffer pointer is {}".format(custom_msg_buffer))
# Casting buffer to Custom Msg Info which will be attached to user meta as CUSTOM_MSG_BLOB
**custom_msg_info = pyds.NvDsCustomMsgInfo.cast(custom_msg_buffer)**
I get this error:
AttributeError: module ‘pyds’ has no attribute ‘NvDsCustomMsgInfo’
Questions:
Does this mean that I have to create bindings for NvDsCustomMsgInfo?
Is this way not supported for Python version of Deepstream?
I haven’t created bindings before and I’m not familiar with pybind11 so I would prefer any easier alternative, but I also don’t want to use the NvDsEventMsgMeta.extMsg way since it has so many additional fields and I need some level of hierarchy in the resulting payload. Is there any other alternative?
yes, from the error, you need to add the binding. please refer to NvDsVehicleObject’s binding binding.
for sending custom data, please refer to topic and /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test4. test4 can send jpeg file by UserMeta.
The topic talks about adding additional metadata through extMsg of NvDsEventMsgMeta. I wanted to avoid NvDsEventMsgMeta because of the additional fields
So if I understand correctly there are 2 options now:
For implementing this solution:
You can attach a custom message blob(string) to payload using NVDS_CUSTOM_MSG_BLOB type by creating NvDsCustomMsgInfo object and specifying the fields message (custom message blob) and len (size of the custom message).
I need to create bindings for the _NvDsCustomMsgInfo C struct (in nvdsmeta.h of deepstream/sources/includes) for pyds so that I can somehow attach it to the user_meta_list of a single frame_meta. I have some questions about this:
Is my logic correct that I need to create a generic buffer using alloc_buffer and then cast it to NvDsCustomMsgInfo? I’m not sure if alloc_buffer will work (documentation is lacking) and I can’t find an alternative.
I do not understand how the NvDsCustomMsgInfo can be attached to the frame_meta.user_meta (assuming one already has NvDsCustomMsgInfo populated).
Do I need to define meta_copy and meta_release functions similarly to the EventMsgMeta case demonstrated in deepstream-test4 or is there another way I should implement?
I haven’t found any samples demonstrating this new-api method which uses NvDsCustomMsgInfo, not even in C deepstream samples.
Can you provide some guidance of what is the correct direction for this?
I do not understand if I have to create a NEW custom binding file in pyds OR modify existing binding of bindnvdsmeta.cpp in pyds to add the NvDsCustomMsgInfo information (constructor fields, allocator, cast etc.).
Adding all my custom metadata to extMsg field of NvDsEventMsgMeta as the topic link you referred to.
My question regarding this:
If I include one single JSON string in extMsg with all my metadata and modify msgconv.cpp source code to only parse extMsg (ignoring other fields like person,sensor,place etc.) is this way equivalent to the custom message blob method in 1. ?
Which of the two methods is preferable for this use-case in pyDS specifically?
alloc_buffer binding is opensource. please refer to the link and deepstream_test_4.py for its usage.
please refer to C version test4. you can use nvds_add_user_meta_to_frame to add.
you can binding a new function alloc_nvds_user_meta for NvDsUserMeta. please refer to alloc_nvds_event_msg_meta.
please refer to opt\nvidia\deepstream\deepstream-6.4\sources\apps\sample_apps\deepstream-test4\dstest4_config.yml, if msg2p-newapi is 1, the app will use new-api. please refer to C version test4 for NvDsCustomMsgInfo usage.
I have spent the last few days investigating this. I migrated to Deepstream6.4, which has the new pyds bindings that, in fact, modify the copy and release functions of NvDsEventMsgMeta in bindschema.cpp.
So, if I understand correctly (correct me if I’m wrong) now there is no way to avoid modifying the pybind11 event_message_copy and release functions & rebuilding the pyds bindings?
How is this an improvement compared to the previous release (user defined copy_func and release_func) in terms of flexibility to define custom user structures?
If that is the case, it is now even more complicated than before to pass custom information to the extMsg field of NvDsEventMsgMeta.
I have also investigated eventmsg_payload.cpp and dsmeta_payload.cpp which are called in nvmsgconv.cpp depending on new-api property.
I don’t understand what is the benefit of using nvmsgconv when having custom json information. I read this topic and this topic from 2021, but it seems that during 3 years releases up to 6.4 there haven’t been any positive developments to support these functionalities in Python Deepstream.
What do you propose as the best (easiest and most reliable) solution to process and send custom metadata to kafka?
if using the latest deepstream_python_apps, the user don’t need to write copy and release for alloc_nvds_event_msg_meta. please find event_msg_meta_copy_func in this link.
I understand that user_meta_copy_func and user_meta_release_func is now deprecated functionality and has moved into the python bindings in event_msg_meta_copy_func.
However, if one wants to use custom functionality (e.g. set extMsg to a string or a CustomDataStruct), I have to modify event_msg_meta_copy_func and event_msg_meta_release_func and then rebuild pyds.
I also have to modify eventmsg_payload.cpp to process my custom extMsg (e.g. parse extMsg string as JsonObject) and rebuild libnvds_msgconv.so (and install it to /opt/nvidia/deepstream/deepstream/lib/ ). This led to this issue which is still unsolved for me.
As a result, I cannot use any of the solutions for custom objects or schema customization.
Therefore, I would like to ask how I can pass custom messages (create custom payload without nvmsgconv) to nvmsgbroker directly, or any alternatives (e.g. use kafka-python directly? What are the disadvantages in performance/asynchronous processing?