Add Frame to MSGBroker payload using EventMetaData

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.3
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) Questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

How should I add a base 64 encoded frame to a user_meta object so that it is sent in my AMQP payload?
I have gotten payloads with the frame data to be sent by using the otherAttrs field and implementing my own eventmsg_payload.cpp file. This gives me the payload I want, but will often result in a segfault.
Here’s how I get the frame.

# Getting Image data using nvbufsurface
# the input should be address of buffer and batch_id
n_frame = pyds.get_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id)

# Convert python array into numpy array format in the copy mode.
frame_copy = np.array(n_frame, copy=True, order='C')
frame_copy = cv2.cvtColor(frame_copy, cv2.COLOR_RGBA2BGRA)
frame_copy = frame_copy.astype(np.uint8)
success, encoded_image = cv2.imencode('.png', frame_copy)
base64_string = base64.b64encode(encoded_image.tobytes()).decode('utf-8')
msg_meta.otherAttrs = base64_string
if is_aarch64(): # If Jetson, since the buffer is mapped to CPU for retrieval, it must also be unmapped 
                pyds.unmap_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id) # The unmap call should be made after operations with the original array are complete.
                                                                                    #  The original array cannot be accessed after this call.

and the eventmsg_payload.cpp code adding the frame

  json_object_set_string_member (jobject, "frame", events[0].metadata->otherAttrs);

I’ve also tried to validate that events[0].metadata != nullptr, no change there though.

I’ve updated meta_copy_func to copy the field

    if srcmeta.otherAttrs:
        dstmeta.otherAttrs = pyds.get_string(srcmeta.otherAttrs)

and updated meta_free_func to free the buffer


I feel confident that this this seg fault is caused by this logic because commenting out the msg_meta.otherAttrs = base64_string line makes my pipeline run consistently without a seg fault.

Is there something else I’m missing that might be causing this segfault? Is otherAttrs a bad place to put such a large piece of data? Could it be causing a buffer overflow or something simliar?

It’s also worth noting I have implemented the accepted answer here. No change.

as you know, DeepStream SDK is a C lib. can you use gdb to debug? can you share the crash stack?

Here is a backtrace.

Thread 35 "queue-queue-fil" received signal SIGSEGV, Segmentation fault.
(gdb) bt
#0  0x00000000004ff12b in  ()
#1  0x0000000000570ba8 in _PyEval_EvalFrameDefault ()
#2  0x00000000005697da in _PyEval_EvalCodeWithName ()
#3  0x00000000005f6ec3 in _PyFunction_Vectorcall ()
#4  0x00000000005f7cb7 in PyObject_CallObject ()
#5  0x00007ffff796fe4e in  () at /usr/lib/python3/dist-packages/gi/
#6  0x00007ffff793be06 in  () at /lib/x86_64-linux-gnu/
#7  0x00007ffff793c188 in  () at /lib/x86_64-linux-gnu/
#8  0x00007ffff3cd245e in  () at /lib/x86_64-linux-gnu/
#9  0x00007ffff77fb996 in g_hook_list_marshal () at /lib/x86_64-linux-gnu/
#10 0x00007ffff3cd1ade in  () at /lib/x86_64-linux-gnu/
#11 0x00007ffff3cd410b in  () at /lib/x86_64-linux-gnu/
#12 0x00007ffff3cd6051 in  () at /lib/x86_64-linux-gnu/
#13 0x00007ffff3cdce63 in gst_pad_push () at /lib/x86_64-linux-gnu/
#14 0x00007ffff391f550 in  () at /lib/x86_64-linux-gnu/
#15 0x00007ffff3cd3fef in  () at /lib/x86_64-linux-gnu/
#16 0x00007ffff3cd6051 in  () at /lib/x86_64-linux-gnu/
#17 0x00007ffff3cdce63 in gst_pad_push () at /lib/x86_64-linux-gnu/
#18 0x00007ffff391f550 in  () at /lib/x86_64-linux-gnu/
#19 0x00007ffff3cd3fef in  () at /lib/x86_64-linux-gnu/
#20 0x00007ffff3cd6051 in  () at /lib/x86_64-linux-gnu/
--Type <RET> for more, q to quit, c to continue without paging--
#21 0x00007ffff3cdce63 in gst_pad_push () at /lib/x86_64-linux-gnu/
#22 0x00007fffc8d92774 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/
#23 0x00007ffff3d0b1e7 in  () at /lib/x86_64-linux-gnu/
#24 0x00007ffff7836384 in  () at /lib/x86_64-linux-gnu/
#25 0x00007ffff7835ae1 in  () at /lib/x86_64-linux-gnu/
#26 0x00007ffff7dac609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#27 0x00007ffff7ee6133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Could you also verify that my approach is reasonable? Is it alright to put the frame into the otherAttrs field to be sent over AMQP?

from the crash stack, the root cause is not clear because it is not related to deepstream code.
to narrow this issue, please remove some code to find which code will cause the crash. python binding and nvmsgconv are opensource. you can add logs to check.

is it h265?

it is related to nvmsgconv. it should not be related to source’s codec.

Is this still an DeepStream issue to support? Thanks!
to narrow down this issue, you can only modify the python code to check if it will crash. especially, you can add log of srcmeta.otherAttrs before pyds.free_buffer(srcmeta.otherAttrs) to check if the content is valid.

Hi, sorry. For some reason these emails got sent to my spam. I’ve just started looking into this again as I got pulled into something else.

I’ve been running lots of tests and I think it might be different from the issue I linked. Even when all of my frame accessing code is commented out, the crash can still happen, although much less often. I have narrowed it down to being a multistream issue. It seems to never happen when I’m processing only one stream. It also never happens if I comment out my probe, so it seems like it’s something in there. I will continue testing and update this branch if I find my problem. If I’m unable to fix the problem, I’ll try to recreate the issue by combining the multistream python app with the frame accessing python app.

Given that it seems to be related to the multistream + probing, does that give you any additional insight?

Are there other components that might need

streammux.set_property("nvbuf-memory-type", mem_type)
nvvidconv.set_property("nvbuf-memory-type", mem_type)

besides video converters and the streammux?

please refer to this code comments.

After digging in more, this seems to be caused by the combination of multistream and msgbroker code. It has nothing to do with getting the image itself. I can get the seg-fault to occur even when the frame_copy code is commented out, which includes adding anything into the otherAttrs.
It also doesn’t happen when only one stream is being processed.

Obviously, seg faults are inconsistent, but it seems like I can get the seg fault to go away by removing the lines below:

pyds.user_copyfunc(user_event_meta, meta_copy_func)
pyds.user_releasefunc(user_event_meta, meta_free_func)

I have two questions.

  1. Do I need to use the above functions if I’m not using extMsg? The README for deepstream app 4 says
  • Provide copy / free functions if meta data is extended through “extMsg” field. *
    which implies that I don’t but I see copies and frees for things like the timestamp as well, which aren’t in extMsg. Clearly, I don’t want to have those fields leaking if I fail to clean them up, so I kept the user_copy and user_release functions, just without the extMsg portions. See below.
user_meta = pyds.NvDsUserMeta.cast(data)
srcmeta = pyds.NvDsEventMsgMeta.cast(user_meta.user_meta_data)

# pyds.free_buffer takes C address of a buffer and frees the memory
# It's a NOP if the address is NULL

 if srcmeta.objSignature.size > 0:
     srcmeta.objSignature.size = 0

If the answer to question 1 is yes, then
2) Should I change anything around how I call these functions when I know my batch size will be greater than 1? It seems like the user_event_meta is attached to the batch_meta, so is it possible that index 2 in the batch overrides index one, causing it to get lost?

Any update on the above?

yes , if using extMsg , you need to implement copy and free functionality. if the meta is copy/free, these copy/free code will be triggered.

what do you mean about " index 2 in the batch overrides index on"?. can you modify deepstream sample to reroduce the issue?

My first question was
Do I need to use the above functions if I’m not using extMsg? In my code, I’m not using extMsg at all. I am using =
msg_meta.bbox.left = obj_meta.rect_params.left
msg_meta.bbox.width = obj_meta.rect_params.width
msg_meta.bbox.height = obj_meta.rect_params.height
msg_meta.trackingId = long_to_uint64(obj_meta.object_id)
msg_meta.confidence = obj_meta.confidence
msg_meta.objClassId = obj_meta.class_id
msg_meta.frameId = frame_number

meta.sensorId = index
meta.placeId = index
meta.moduleId = index
meta.sensorStr = f"sensor-{index}"
meta.componentId = index
meta.ts = pyds.alloc_buffer(MAX_TIME_STAMP_LEN + 1)

If I do need the copy/free funcs, then I can try to reproduce this issue in a test app.

The second question is genuinely just spitballing as to why multistream might be causing issues. I didn’t know if different streams were stepping on each others toes or something along those lines.

  1. yes, the app needs to know how to copy/free user_event_meta.
  2. I did not encounter this kind of problem. please modify deepstream sample to reproduce the issue if you encountered, Thanks!

Ok. I will try to reproduce in a test app.

Also, I think it’s worth updating this part of the README.

Provide copy / free functions if meta data is extended through “extMsg” field. 

To me, this implies that these functions aren’t needed unless extMsg is used.

1 Like

Thanks for the sharing!

Is this still an DeepStream issue to support? Thanks!

Yes. I’ve been getting pulled into other stuff. Sorry. Is there any advice you can give on debugging with gdb? Are there any debug symbols I need to download or anything similar?

if crashed in DeepStream opensource code or custom code, you can rebuild the code with “-g” flag, then use gdb “bt” to get the full crash stack, then analyze and simplify code to narrow the issue. if crashed in DeepStream closedsource code, please share the reprodcuing steps. no need other symbols to download.