Deepstream core dump when running with a hikivision camera

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.1
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.2.5-1+cuda11.4
• NVIDIA GPU Driver Version (valid for GPU only) 510.68.02
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
When running deepstream-app with hikivision camera，i set pipeline-recreate-sec=5 . And after 5 seconds, the pipeline would recreate . But there comes a question, when i run with h264 file, the pipelien can recreate correctly. When i set 1 hikivision camera, the pipeline recreate raised core dump at gst_element_get_static_pad.
Here is the bt info:

bca13093a7ce655440361f02f4d5befe
bca13093a7ce655440361f02f4d5befe1084×722 110 KB

Here is the debug log with --gst-debug-level=4:

image
image1208×751 128 KB

Here is the config file:
deepstream_app_config_rtsp.txt (3.5 KB)

Now i have only one hikivision camera, so i can`t test different type camera.

3
  1. can the hikivision camera be played ok by thirdpart tools like ffplay or VLC?
  2. To narrow down this issue, please only use one source and one sink, and share the whole terminal loglevel 6 log, thanks!.
4
  1. it can played by vlc
  2. I test with one hikivision camera ,Here is the loglevel 6 log:
    hikivision.log (91.3 MB)
5
  1. if use one source and one sink, dose deepstream-app crashed again?
  2. from the logs, there is rtspserver error “rtspserver rtsp-server.c:928:gst_rtsp_server_create_socket:GstRTSPServer@0x557e7739b640e[00m failed to create socket”,
    can you try sink type=3? please refer to source30_1080p_dec_infer-resnet_tiled_display_int8.txt, it should not the camera’s issue if sink type=3 works.
  3. please try different rtsp-port if type=4 because binding port failed in the logs.
6
  1. Only one file source and one sink can recreate normally. One rtsp source and one file sink will core dump at gst_element_get_static_pad like aforementioned.
    Here is the config file:
    [application]
    enable-perf-measurement=1
    perf-measurement-interval-sec=5

[tiled-display]
enable=0
rows=1
columns=1
width=1280
height=720
gpu-id=0
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP 5=CSI
type=4
#uri=file:/opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_1080p_h264.mp4
num-sources=1
uri=rtsp://admin:zhy12345@172.16.72.64:554/Streaming/Channels/101
gpu-id=0
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay
type=3
source-id=0
container=1
sync=0
gpu-id=0
nvbuf-memory-type=0
codec=1
enc-type=0
qos=0
bitrate=4000000
iframeinterval=30
output-file=out.mp4

[osd]
enable=1
gpu-id=0
border-width=5
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=1
batched-push-timeout=40000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0

[primary-gie]
enable=1
gpu-id=0
batch-size=4
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_yoloV5.txt

[tests]
file-loop=1
pipeline-recreate-sec=10

  1. When i set a rtsp source and a file sink ， it also make a segfault.
    Here is the log:
    oneHikivisionOneFileSink.zip (9.1 MB)

  2. I found a new question, running with a file source and a file sink, i set pipeline-recreate-sec=5, it will core dump without gdb. When i gdb the deepstream-app with pipeline-recreate-sec=5 ,it can run normally. And running with –gst-debug-level=4, it also can run normally.
    So i edit pipeline-recreate-sec=10, it can run normally without gdb. Does the pipeline-recreate-sec influence the recreate result? In other word, does this means i can`t recreate the pipeline frequently? Or it may lead to some mistake.

7
  1. please set engine path in config_infer_primary_yoloV5.txt, otherwise it will create engine every time it runs.
  2. if did not set pipeline-recreate-sec, dose “set a rtsp source and a file sink” run ok, I found error “gstrtspsrc.c:6563:gst_rtspsrc_send:<src_elem0>e[00m error: Option not supported (551)” in the log oneHikivisionOneFileSink.zip.

I can’t reproduce this issue by deepstream native Primary_Detector model, here is the cfg:
c.txt (1.3 KB)

8
  1. i set model_engine_file in config_infer_primary_yolov5.txt
  2. Set hikivision rtsp source and a file sink, the pipeline can run normally. I google the error “Option not supported(551)”, it says the hikivision don`t support pause ability. When invoke gst_element_set_state(PAUSED), it will send the pause command to the camera.
  3. I will test this question more times.
9

Sorry for the late reply, Is this still any DeepStream issue to support? Thanks

10

yes , i found when i set the source type to 3 means multiuri (with one rtsp source and one sink), it can recreate the pipeline normally.

11

multiuri is not rtsp type.
from log oneHikivisionOneFileSink.zip, there is “Recreate pipeline” but no “can’t set pipeline to playing state”, please add more logs in recreate_pipeline_thread_func to check which line will trigger crash.

12

From log oneHikivisionOneFileSink.zip, there is “Recreate pipeline” but no “can’t set pipeline to playing state”, it crashed at the first recreate, from the crash stack, it crashed in cb_newpad of deepstream_source_bin.c, it seemed one thread is using object while other thread is destroying. why do you need to set pipeline-recreate-sec=5, please set it to 10 to work around.

13

Cause i wrap the deepstream with grpc, thus the program can interact with a java web service. It can start to ai-analysis when java service send a start command . And it can stop ai-analysis when java service send a stop command.
To implement this command, i refer to the recreate_pipeline_thread_func. When start ai-analysis, i invoke create_pipeline and set the pipeline to playing state. When stop ai-analysis, i invoke destroy_pipeline_func.
To ensure the interface work normally, i should test the recreate_pipeline_thread_func first.

14

I will test this mean 2 days later cause we are in spring festival. The camera in the company has no power.

15

I set pipeline-recreate-sec=10 and pipeline-recreate-sec=20 , it comes the same problem like pipeline-recreate-sec=5.

16

Using one rtsp source(not hikivision) and one fakesink, I can reproduce this issue, but the stack is not in cb_newpad3, will continue to check.
source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt (5.7 KB)
stack.txt (3.5 KB)

After adding logs in cb_rtspsrc_select_stream, at the first time, it will create bin->depay, after recreating pipeline, it did not enter cb_rtspsrc_select_stream, so the application crashed in cb_newpad3 because bin->depay is invalid.

18

root cause:
AppCtx is global variable, NvDsSrcBin’s depay is not set to NULL after destroying pipeline, NvDsSrcBin’s depay will not be created again after new creating pipeline, as a result, the application crashed in cb_newpad3 because bin->depay is invalid.

fix:
set NvDsSrcBin’s depay to NULL in destroy_pipeline.

19

I make a simple test, if the bin->depay is old. It can also invoke gst_element_get_static_pad（） correctly. So Why the program core dump at gst_ele_get_static_pad funcion?
Here is the test code:
loop = g_main_loop_new(NULL, FALSE);

gst_bus_add_watch(GST_ELEMENT_BUS(pipeline), bus_cb, loop);

g_timeout_add_seconds(5, timeout_cb, loop);

g_main_loop_run(loop);

gst_element_set_state(pipeline, GST_STATE_NULL);

if(q1){
g_print("q1 address: %d \n",q1);
}
else{
	g_print("q1 NULL");
}
GstElement* test_queue = gst_element_factory_make("queue", "test_queue");
if(test_queue){
	g_print("test_queue  addr:%d \n",test_queue);
}
GstPad *sinkpad = gst_element_get_static_pad (q1, "sink");
if(sink){
	g_print("get pad success \n");
}
gst_object_unref(pipeline);

In the time_cb , the loop will break.

20

with this fix “set NvDsSrcBin’s depay to NULL in destroy_pipeline”, will your code crash again(RTSP source)?
after destroying pipeline, NvDsSrcBin’s depay will be freed, but it was not set to NULL, so it is a wild pointer.

21

Another quesiton, i want to know how to trigger the rtspsrc select-stream signal.The doc says “Emitted before the client decides to configure the stream num with caps.” But in deepstream , i can`t find any code about set caps to rtspsrc element. Thanks.

22

I will test tomorrow, the camera is at company. I can`t connect it now. Wild pointer, I know. --!