Deadlock issue with NVV4l2Decoder

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

  • Hardware Platform (Jetson / GPU) - ZOTAC GeForce RTX 2060 SUPER

  • DeepStream Version - v5.0.1

  • JetPack Version (valid for Jetson only) - N/A

  • TensorRT Version - N/A

  • Issue Type (questions, new requirements, bugs) - Bug

  • How to reproduce the issue?

This issue can be reproduced using a number of H264 encoding profiles. For now, the following H264 encoding profiles appear to cause issues:

  • “high-4:4:4”

  • “high-4:2:2”

  • “high-10”

  • “high-4:4:4-intra”

  • “high-4:2:2-intra”

  • “high-10-intra”

This issue occurs consistently regardless of the input method (local video file or RTSP stream). Therefore, one way to reproduce this bug is shown below:


gst-launch-1.0 videotestsrc num-buffers=900 ! video/x-raw,framerate=15/1,width=800,height=600 ! x264enc ! video/x-h264,profile=high-4:4:4 ! h264parse ! mp4mux ! filesink location=/tmp/test.mp4

gst-launch-1.0 filesrc location=/tmp/test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

Upon attempting to decode a H264 stream using one of the above encoding profiles, the NVV4l2Decoder element blocks and will not be able to accept any more incoming buffers from linked elements; nor will it output any buffers. In addition, attempting to set the NVV4l2Decoder element’s state to NULL at this point will result in the call to gst_element_set_state deadlocking.

According to gdb, the current state of the threads when attempting to cancel the pipeline in gst-launch-1.0 is posted below:


Thread 7 (Thread 0x7fffe17fe700 (LWP 288)):

#0 0x00007ffff731d9f3 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0

#1 0x00007ffff28b8a6b in NvOsSemaphoreWait(NvOsSemaphoreRec*) ()

from ///opt/nvidia/deepstream/deepstream-5.0/lib/libcuvidv4l2.so

#2 0x00007ffff28caf75 in cuvidv4l2_dec_thread_func(void*) ()

from ///opt/nvidia/deepstream/deepstream-5.0/lib/libcuvidv4l2.so

#3 0x00007ffff28b8d16 in thread_wrapper(void*) () from ///opt/nvidia/deepstream/deepstream-5.0/lib/libcuvidv4l2.so

#4 0x00007ffff73176db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

#5 0x00007ffff7040a3f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7fffe1fff700 (LWP 287)):

#0 0x00007ffff731df85 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0

#1 0x00007ffff135b1bf in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1

#2 0x00007ffff1454606 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1

#3 0x00007ffff73176db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

#4 0x00007ffff7040a3f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7fffe8eb2700 (LWP 286)):

#0 0x00007ffff7033cf9 in poll () from /lib/x86_64-linux-gnu/libc.so.6

#1 0x00007ffff145e551 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1

#2 0x00007ffff14375fa in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1

#3 0x00007ffff1454606 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1

#4 0x00007ffff73176db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

#5 0x00007ffff7040a3f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x7fffe98e6700 (LWP 285)):

#0 0x00007ffff7033cf9 in poll () from /lib/x86_64-linux-gnu/libc.so.6

#1 0x00007ffff757b5c9 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0

#2 0x00007ffff757b6dc in g_main_context_iteration () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0

#3 0x00007ffff757b721 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0

#4 0x00007ffff75a3175 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0

#5 0x00007ffff73176db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

#6 0x00007ffff7040a3f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7fffea0e7700 (LWP 284)):

#0 0x00007ffff7003a30 in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6

#1 0x00007ffff7036977 in usleep () from /lib/x86_64-linux-gnu/libc.so.6

#2 0x00007ffff40cd6df in gst_v4l2_video_dec_handle_frame ()

from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so

#3 0x00007ffff52c8ea1 in ?? () from /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0

#4 0x00007ffff52cb994 in ?? () from /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0

#5 0x00007ffff52cc072 in ?? () from /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0

#6 0x00007ffff7b1088b in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#7 0x00007ffff7b18bb3 in gst_pad_push () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#8 0x00007ffff5c48728 in gst_base_parse_push_frame () from /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0

#9 0x00007ffff5c4b28b in gst_base_parse_finish_frame () from /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0

#10 0x00007ffff4539e5e in ?? () from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvideoparsersbad.so

#11 0x00007ffff5c435a2 in ?? () from /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0

#12 0x00007ffff5c4943e in ?? () from /usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0

#13 0x00007ffff7b1088b in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#14 0x00007ffff7b18bb3 in gst_pad_push () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#15 0x00007ffff59be966 in ?? () from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstisomp4.so

#16 0x00007ffff59dc4c5 in ?? () from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstisomp4.so

#17 0x00007ffff7b45269 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#18 0x00007ffff75a3b40 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0

#19 0x00007ffff75a3175 in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0

#20 0x00007ffff73176db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

---Type <return> to continue, or q <return> to quit---

#21 0x00007ffff7040a3f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7fffeab04700 (LWP 283)):

#0 0x00007ffff70423e7 in accept4 () from /lib/x86_64-linux-gnu/libc.so.6

#1 0x00007ffff13b24a3 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1

#2 0x00007ffff1454606 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1

#3 0x00007ffff73176db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

#4 0x00007ffff7040a3f in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7ffff7fdf740 (LWP 279)):

#0 0x00007ffff732111d in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0

#1 0x00007ffff731a098 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0

#2 0x00007ffff7b1486f in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#3 0x00007ffff7b15325 in gst_pad_set_active () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#4 0x00007ffff7af2f0d in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#5 0x00007ffff7b05874 in gst_iterator_fold () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#6 0x00007ffff7af3a16 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#7 0x00007ffff7af595e in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#8 0x00007ffff7af5c06 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#9 0x00007ffff52ce365 in ?? () from /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0

#10 0x00007ffff40ceb53 in gst_v4l2_video_dec_change_state ()

from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so

#11 0x00007ffff7af7d5e in gst_element_change_state () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#12 0x00007ffff7af8499 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#13 0x00007ffff7ad5a02 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#14 0x00007ffff7af7d5e in gst_element_change_state () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#15 0x00007ffff7af8499 in ?? () from /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0

#16 0x0000555555557ace in ?? ()

#17 0x00007ffff6f40b97 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6

#18 0x00005555555580da in ?? ()

In particular, it appears that gst_v4l2_video_dec_change_state is deadlocking due to gst_v4l2_video_dec_handle_frame also being in a perpetually deadlocked state.

Is it possible for the NVV4L2Decoder element to raise an error if it receives H264 data with an unsupported encoding profile, so that we do not get stuck in a deadlocked state?

3

We can reproduce the failure. Will look into it.

5

Hello, Fiona!
Have you found any solution to this problem yet?

6

any solution? I have the same issue.

Thread 8 (Thread 0x7f94e1a98700 (LWP 63449)):
#0  0x00007f950c2d91fd in __lll_lock_wait () at /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f950c2d20f4 in pthread_mutex_lock () at /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f950ccf387f in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#3  0x00007f950ccf4335 in gst_pad_set_active () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#4  0x00007f950ccd1f0d in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#5  0x00007f950cce4884 in gst_iterator_fold () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#6  0x00007f950ccd2a16 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#7  0x00007f950ccd495e in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#8  0x00007f950ccd4c06 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#9  0x00007f94b79a4365 in  () at /usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#10 0x00007f9430169f41 in gst_v4l2_video_dec_change_state () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
#11 0x00007f950ccd6d5e in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#12 0x00007f950ccd7499 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#13 0x00007f950ccb4a02 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#14 0x00007f946d30a029 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#15 0x00007f950ccd6d5e in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#16 0x00007f950ccd7499 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#17 0x00007f950ccb4a02 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#18 0x00007f946d31d69a in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#19 0x00007f950ccd6d5e in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#20 0x00007f950ccd7499 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#21 0x00007f950ccb4a02 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#22 0x00007f950ccd6d5e in gst_element_change_state () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#23 0x00007f950ccd7499 in  () at /usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#24 0x00005579ae0776f5 in deepstream::PipeLine::remove_source_bin(unsigned int, bool) (this=0x7f94bc000d30, index=27, lock=false) at /root/ZLMediaKit/server/ds_pipeline.cpp:890
#25 0x00005579ae077d53 in deepstream::PipeLine::stop_task(unsigned int) (this=0x7f94bc000d30, index=27) at /root/ZLMediaKit/server/ds_pipeline.cpp:967
#26 0x00005579ae0792dc in deepstream::PipeLine::check_stream() (this=0x7f94bc000d30) at /root/ZLMediaKit/server/ds_pipeline.cpp:1130
#27 0x00005579ae0598a5 in deepstream::WorkerThread::process() (this=0x7f94b8001040) at /root/ZLMediaKit/server/ds_WorkerThread.cpp:357
#28 0x00005579ae05bec8 in std::__invoke_impl<void, void (deepstream::WorkerThread::*)(), deepstream::WorkerThread*>(std::__invoke_memfun_deref, void (deepstream::WorkerThread::*&&)(), deepstream::WorkerThread*&&) (__f=@0x7f94b8001980: (void (deepstream::WorkerThread::*)(deepstream::WorkerThread * const)) 0x5579ae0590ca <deepstream::WorkerThread::process()>, __t=@0x7f94b8001978: 0x7f94b8001040) at /usr/include/c++/7/bits/invoke.h:73
#29 0x00005579ae05afab in std::__invoke<void (deepstream::WorkerThread::*)(), deepstream::WorkerThread*>(void (deepstream::WorkerThread::*&&)(), deepstream::WorkerThread*&&) (__fn=@0x7f94b8001980: (void (deepstream::WorkerThread::*)(deepstream::WorkerThread * const)) 0x5579ae0590ca <deepstream::WorkerThread::process()>) at /usr/include/c++/7/bits/invoke.h:95
#30 0x00005579ae05e5a9 in std::thread::_Invoker<std::tuple<void (deepstream::WorkerThread::*)(), deepstream::WorkerThread*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x7f94b8001978) at /usr/include/c++/7/thread:234
#31 0x00005579ae05e54e in std::thread::_Invoker<std::tuple<void (deepstream::WorkerThread::*)(), deepstream::WorkerThread*> >::operator()() (this=0x7f94b8001978)
    at /usr/include/c++/7/thread:243
---Type <return> to continue, or q <return> to quit---

#32 0x00005579ae05e520 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (deepstream::WorkerThread::*)(), deepstream::WorkerThread*> > >::_M_run() (this=0x7f94b8001970)
    at /usr/include/c++/7/thread:186
#33 0x00007f9506b2f6df in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#34 0x00007f950c2cf6db in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#35 0x00007f95061ec71f in clone () at /lib/x86_64-linux-gnu/libc.so.6
7

@Fiona.Chen

8

Hello @xastra This issue has been fixed in DS6.1, you can upgrade to DS6.1 and check it, thank you.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.