Deadlock in nvv4l2decoder

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.2
• NVIDIA GPU Driver Version (valid for GPU only)
525.105.17
• Issue Type( questions, new requirements, bugs)
bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I ran myapp with a bad rtsp stream, which produced errors like these:

WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
ERROR from depay_elem0: The stream is in the wrong format.
Debug info: gstrtph264depay.c(1298): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
NAL unit type 27 not supported yet
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
[Thread 0x7f270b36d000 (LWP 64) exited]
[Thread 0x7f270ab6c000 (LWP 65) exited]
** INFO: <reset_source_pipeline:1707>: Resetting source 0
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
ERROR from depay_elem0: The stream is in the wrong format.
Debug info: gstrtph264depay.c(1298): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
NAL unit type 27 not supported yet
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
ERROR from depay_elem0: The stream is in the wrong format.
Debug info: gstrtph264depay.c(1298): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
NAL unit type 27 not supported yet
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
[New Thread 0x7f270b36d000 (LWP 67)]
[New Thread 0x7f270ab6c000 (LWP 68)]
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
ERROR from depay_elem0: The stream is in the wrong format.
Debug info: gstrtph264depay.c(1298): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
NAL unit type 27 not supported yet
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
ERROR from depay_elem0: The stream is in the wrong format.
Debug info: gstrtph264depay.c(1298): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
NAL unit type 27 not supported yet
WARNING from depay_elem0: Could not decode stream.
Debug info: gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
Undefined packet type
ERROR from depay_elem0: The stream is in the wrong format.
Debug info: gstrtph264depay.c(1298): gst_rtp_h264_depay_process (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstRtpH264Depay:depay_elem0:
NAL unit type 27 not supported yet

but when I tried to stop it by calling gst_element_set_state (bin->bin, GST_STATE_NULL);, it stucked. I used gdb to debug, backtrace like this:

#0  0x00007fffd97fa170 in __lll_lock_wait () at /usr/lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fffd97f2131 in pthread_mutex_lock () at /usr/lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007fffd98f28ea in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#3  0x00007fffd98f3035 in gst_pad_set_active () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#4  0x00007fffd98cfd25 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#5  0x00007fffd98e2d3c in gst_iterator_fold () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#6  0x00007fffd98d0546 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#7  0x00007fffd98d255e in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#8  0x00007fffd98d2791 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#9  0x00007fffcc33553d in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#10 0x00007fffd98d49d2 in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#11 0x00007fffd98d5119 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#12 0x00007fffd98b11b8 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#13 0x00007fff10be59ba in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#14 0x00007fffd98d49d2 in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#15 0x00007fffd98d5119 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#16 0x00007fffd98b11b8 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#17 0x00007fffd98d49d2 in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

there’s a lock waiting, I traced the lock’s oewer thread, which is this one:

#0  0x00007fffd93a123f in clock_nanosleep () at /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fffd93a6ec7 in nanosleep () at /usr/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fffd93d985f in usleep () at /usr/lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fff10a31c21 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
#4  0x00007fffcc32fdab in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#5  0x00007fffcc332af8 in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#6  0x00007fffcc3331ea in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#7  0x00007fffd98ecfef in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#8  0x00007fffd98ef051 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#9  0x00007fffd98f5e63 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#10 0x00007fffd926f550 in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#11 0x00007fffd98ecfef in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#12 0x00007fffd98ef051 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#13 0x00007fffd98f5e63 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#14 0x00007fffd92532c7 in gst_base_parse_push_frame () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#15 0x00007fffd925603b in gst_base_parse_finish_frame () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#16 0x00007fff10b13fff in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvideoparsersbad.so
#17 0x00007fffd924dde6 in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#18 0x00007fffd9253f2e in  () at /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#19 0x00007fffd98ecfef in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#20 0x00007fffd98ef051 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#21 0x00007fffd98f5e63 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#22 0x00007fffd98ecfef in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#23 0x00007fffd98ef051 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#24 0x00007fffd98f5e63 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#25 0x00007fffd98dbd8b in gst_proxy_pad_chain_default () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#26 0x00007fffd98ecfef in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#27 0x00007fffd98ef051 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#28 0x00007fffd98f5e63 in gst_pad_push () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#29 0x00007fffcc16a774 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstcoreelements.so
#30 0x00007fffd99241e7 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#31 0x00007fffd9a35374 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#32 0x00007fffd9a34ad1 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#33 0x00007fffd97ef609 in start_thread () at /usr/lib/x86_64-linux-gnu/libpthread.so.0
#34 0x00007fffd93e3133 in clone () at /usr/lib/x86_64-linux-gnu/libc.so.6

seemed like it stucked in usleep(), I tried continue and break several times, each time it stopped exactly the same position.
I tried deepstream-app in the nvcr.io/nvidia/deepstream:devel-6.2 with no code changing, it also stucked when I tried to stop the application, and from the gdb backtrace, it occured when calling gst_element_set_state () just like my own app:

#0  0x00007f1eb3968170 in __lll_lock_wait () at /usr/lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f1eb3960131 in pthread_mutex_lock () at /usr/lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f1eb422f8ea in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#3  0x00007f1eb4230035 in gst_pad_set_active () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#4  0x00007f1eb420cd25 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#5  0x00007f1eb421fd3c in gst_iterator_fold () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#6  0x00007f1eb420d546 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#7  0x00007f1eb420f55e in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#8  0x00007f1eb420f791 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#9  0x00007f1eb434c53d in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#10 0x00007f1eb42119d2 in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#11 0x00007f1eb4212119 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#12 0x00007f1eb41ee1b8 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#13 0x00007f1e6cc2d9ba in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#14 0x00007f1eb42119d2 in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#15 0x00007f1eb4212119 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#16 0x00007f1eb41ee1b8 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#17 0x00007f1eb42119d2 in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#18 0x00007f1eb4212119 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#19 0x0000556138f7a5a9 in reset_source_pipeline (data=0x7f1e7a448088) at ../../apps-common/src/deepstream_source_bin.
#20 0x0000556138f75939 in watch_source_status (data=0x7f1e7a448088) at ../../apps-common/src/deepstream_source_bin.c:
#21 0x00007f1eb3f10be8 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#22 0x00007f1eb3f1004e in g_main_context_dispatch () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#23 0x00007f1eb3f10400 in  () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#24 0x00007f1eb3f106f3 in g_main_loop_run () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#25 0x0000556138f64195 in main (argc=1, argv=0x7ffcac7ea978) at deepstream_app_main.c:824
1 Like

What do you mean a bad rtsp stream? Could you share the rtsp link to us?

The rtsp stream is an internal address,I’m afraid it can’t be accessed from out network.This stream has some data error and cannot be played.

OK. Then could you dump the stream to a file and send to us?

I used gstreamer and saved part of the stream.Hope that would be helpful.

gst-launch-1.0 rtspsrc location="rtsp://" ! filesink location=./stream

stream (6.2 MB)

We cannot use this file to analyze. Could you save it to h264 format or mp4 file?

No I couldn’t, I tried ffmpeg and gstreamer and got errors.The point is I can’t control what kind of rtsp streams I will receive from server, and if it’s corrupted in some way, I don’t want my program stucked.

OK. If you couldn’t provide the h264 or mp4 streams. You can try it after our next version is released. We have fixed some similar issues like yours, like 242132

Ok,I’ll wait then.

I have the same issue. May I ask when the problem will be fixed / next version is released? Is there any workaround?

Could you attach your video source? The issue may be the same, but the root cause may be different. Or you can just wait for the update of next version, it’ll be soon.

I ma unable to share the video source with you. I have GStreamer pipeline with dynamic source add / removal functionality based on:

The problem occurs when the rtsp video source goes offline randomly and I try to reinitialize the source by removing it and adding it again. The most of the times the reinitialization works but sometimes it deadlocks when changing the source bin element to state GST_STATE_NULL when removing it:

gst_element_set_state (binElement, GST_STATE_NULL);

this is a part of GDB bt of the thread calling gst_element_set_state:

Thread 11 (Thread 0x7fc024993700 (LWP 20876)):
#0  0x00007fc0c25481fd in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fc0c25410f4 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007fbcd6f71ab7 in gst_rtspsrc_change_state (element=0x7fbcb171f920, transition=GST_STATE_CHANGE_PLAYING_TO_PAUSED) at ../gst/rtsp/gstrtspsrc.c:8490
#3  0x00007fc0c3f8fd5e in gst_element_change_state () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#4  0x00007fc0c3f90499 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#5  0x00007fc0c3f6da02 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#6  0x00007fc0c3f8fd5e in gst_element_change_state () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#7  0x00007fc0c3f90499 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

The issue is similar to:

When running my app with GST_DEBUG=3 i get this error when the deadlock occurs:

ESC[0m0:01:47.962647686 ESC[335m171634ESC[00m 0x7fff418a6190 ESC[31;01mERROR  ESC[00m ESC[00m       v4l2allocator gstv4l2allocator.c:1384:gst_v4l2_allocator_qbuf:<nvv4l2decoder66:pool:src:allocator>ESC[00m faile
d queueing buffer 2: Bad file descriptor
0:01:47.962658526 ESC[335m171634ESC[00m 0x7fff418a6190 ESC[31;01mERROR  ESC[00m ESC[00m      v4l2bufferpool gstv4l2bufferpool.c:1405:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder66:pool:src>ESC[00m could not queue a 
buffer 2
0:01:47.962672363 ESC[335m171634ESC[00m 0x7fff418a6190 ESC[33;01mWARN   ESC[00m ESC[00m       v4l2allocator gstv4l2allocator.c:838:gst_v4l2_allocator_stop:<nvv4l2decoder66:pool:src:allocator>ESC[00m error releas
ing buffers buffers: Bad file descriptor
0:01:48.149386134 ESC[335m171634ESC[00m 0x7fff418a6190 ESC[31;01mERROR  ESC[00m ESC[00m       v4l2allocator gstv4l2allocator.c:1384:gst_v4l2_allocator_qbuf:<nvv4l2decoder67:pool:src:allocator>ESC[00m failed queu
eing buffer 1: Bad file descriptor
0:01:48.149414558 ESC[335m171634ESC[00m 0x7fff418a6190 ESC[31;01mERROR  ESC[00m ESC[00m      v4l2bufferpool gstv4l2bufferpool.c:1405:gst_v4l2_buffer_pool_qbuf:<nvv4l2decoder67:pool:src>ESC[00m could not queue a 
buffer 1

If it’s an emergency for you, you can try to decode with other decoders(for example, ffmpeg), and send decoded frames to your pipeline by appsrc, there are some example about how to use appsrc in deepstream-app(deepstream-appsrc-cuda-test). It works fine with me anyway. And it’s really hard for them to reproduce and solve such problems if they can’t get the same source.I hope it will help you.

yes, the appsrc could be a solution, but it would require quite a huge redesign for my application…the problem occurs on any source when intesive source add / removal occurs or many reconnections occur if the source is offline

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Just from the GDB log you attached, it’s locked in the native Gstreamer plugin. We cannot further analyze the problem. You can try the following 2 methods:

  • You can use your source to test the deepstream-app, this demo has perfect reconnection mechanism. If there are no issues, you can try adding a reconnection mechanism according to our open source code:check_rtsp_reconnection_attempts in sources\apps\apps-common\src\deepstream_source_bin.c.
  • You can wait for our next update, it’ll be soon.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.