Segmentation Fault in DeepStream Kafka Pipeline During Metadata Copy (Exit Code 139)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform GPU
• DeepStream Version 6.1
• TensorRT Version 8.2.5
• NVIDIA GPU Driver Version (valid for GPU only) 560.35.03
• Issue Type( questions, bugs)
**• I have built a custom Python DeepStream Kafka pipeline that works successfully with three RTSP sources. However, when any of the RTSP streams go down or disconnect, the DeepStream pipeline restarts, and I encounter a segmentation fault, causing the program to stop.
i am trying to debug this issue, I found that commenting out the following lines in my code prevents the segmentation fault and pipeline restart, but also causes the Kafka consumer to stop receiving messages:

python code
user_event_meta.user_meta_data = msg_meta
user_event_meta.base_meta.meta_type = pyds.NvDsMetaType.NVDS_EVENT_MSG_META
pyds.user_copyfunc(user_event_meta, meta_copy_func)
pyds.user_releasefunc(user_event_meta, meta_free_func)
pyds.nvds_add_user_meta_to_frame(frame_meta, user_event_meta)
pyds.register_user_releasefunc(meta_free_func)

With these lines commented , the pipeline runs fine, but no messages are being received by the Kafka consumer.
i dont know why this is happening and how to fix the issue so that the Kafka consumer receives messages without causing the segmentation fault?

3

which sample are you testing or referring to? could you share a whole log? wondering if there is any error tip. can you reproduce this issue on the latest Deepstream 7.0?

4

I am using the Docker image of this nvcr.io/nvidia/deepstream:6.1-base, and I am referring to deepstream test 4.
pipeline.py.txt (29.7 KB)
This is my pipeline code for referee is deepstream test4.

5

Thanks for the sharing! seems there are many custom code in your code. can the native deepstream_test_4.py run well on your device? if so, could you simplify your code to check which part code cause the crash?

6

Yes, I am trying deepstream_test_4.py, and it’s working fine. I am adding a monitor class to monitor if the RTSP links are down, then restart the deepstream pipeline when I get fps 0, 0, 0 and below code line ‘’’

msg_meta = pyds.alloc_nvds_event_msg_meta()

msg_meta=generate_event_msg_meta(obj_meta, frame_meta,user_meta_data,self.source_dict[source_id],self.main_key,n_frame,source_id)

deln_frame

obj_list.add(obj_meta.object_id)

user_event_meta=pyds.nvds_acquire_user_meta_from_pool(batch_meta)

user_event_meta.user_meta_data =msg_meta

user_event_meta.base_meta.meta_type =pyds.NvDsMetaType.NVDS_EVENT_MSG_META

ifframe_metaanduser_event_meta:

try:

pyds.nvds_add_user_meta_to_frame(frame_meta, user_event_meta) #send to kafka producer

logging.info(“Successfully added user meta to frame”)

exceptExceptionase:

logging.error(f"Error adding user meta to frame: {str(e)}")

else:

logging.warning(“frame_meta or user_event_meta is None”)

else:

logging.warning(“Error in attaching event meta to buffer”)

pyds.user_copyfunc(user_event_meta, meta_copy_func)

pyds.user_releasefunc(user_event_meta, meta_free_func)

‘’’

my docker container crash

7

and below are my monitor.py script
monitor.py.txt (3.4 KB)

8

why is “stop pipeline” not called in monitor.py? if disabling sending broker, will the app crash again?
please refer to this topic. why not use RTSP reconnection feature instead of restarting the whole pipeline?

9

after add uri_decode_bin=Gst.ElementFactory.make(“nvurisrcbin”, “uri-decode-bin”)
uri_decode_bin.set_property(“rtsp-reconnect-interval”, 5)
and testing restart rtsp stream got this output


inference are stuck in last frame and current frame

10
  1. DeepStraem 6.1 requires Display Driver R510.47.03, please refer to this table and install DeepStream by this doc.
  2. On DS7.0, after adding the code modification mentioned on OCT 21 based on test3, the app run well and the output video is fine. Here is the log: log-1023.txt (5.0 KB)
11

I am debugging this issue and found "nvurisrcbin.” When I set uridecodebin, my pipeline works well. I am running deepstream_test1_rtsp_in_rtsp_out.py. When I set uridecodebin pipeline and stream, they are working well, but when I set nvurisrcbin, I got disrupted RTSP stream, and this issue is in DS6.1, but when I run on DS7.0, it is working well.

12

Thanks for the sharing! please use DS7 if the latter versions work.

13

But this plugin also supports DS6.1, and my other requirements require the Ubuntu 20 os. I am unable to transition to DS7.0. Please tell me the solution to DS6.1.

14

Let 's back to your original issue. please refer to the latest code. user_copyfunc and user_releasefunc are not needed because there are moved to alloc_nvds_event_msg_meta. DeepStream Python code is opensource. you can port this simplified code. especially you need to rebuild code and reinstall whl after modifying the code. please refer to this doc.

15

after add and build pyds i got below error ,
ERROR:root:Error in tiler sink buffer probe : alloc_nvds_event_msg_meta(): incompatible function arguments. The following argument types are supported:
1. (arg0: pyds.NvDsUserMeta) → pyds.NvDsEventMsgMeta
I am build pyds version 1.1.3 for deepstream 6.1

16

seems the parameter is wrong. please refer to this sample code.

17

I changed the bindschema.cpp file as you instructed, built pyds, and added the following line: msg_meta = pyds.alloc_nvds_event_msg_meta(user_event_meta).
I received this error: TypeError: incompatible function arguments in alloc_nvds_event_msg_meta(). The sorts of arguments supported are as follows:

  1. ->pyds.NvDsEventMsgMeta (arg0: pyds.NvDsUserMeta)
18
  1. please refer to this code. alloc_nvds_event_msg_meta requires parameter “NvDsUserMeta *user_meta” and returns NvDsEventMsgMeta. please find alloc_nvds_event_msg_meta in all code and make sure the related code are consistent.
  2. after rebuilding, please reinstall whl file. here is a smaple.
pip3 uninstall    xx.whl  && pip3 install    xx.whl