We are observing an abort coming from within nvv4l2decoder
/libgstnvvideo4linux2.so
when using gst discoverer on an h264 encoded file.
Reproduction script tested on 2 jetson boxes:
- Orin NX:
# R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug 1 19:57:35 UTC 2023
- Xavier NX:
# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020
#prepare bad test file
gst-launch-1.0 videotestsrc num_buffers=13 ! video/x-raw, framerate=15/1, width=1920, height=1080 ! nvvidconv ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=./test_bad.mkv
# try in a loop until a crash
while true
do
gst-discoverer-1.0 ./test_bad.mkv || break
done
Sometimes the loop will run for a few seconds before crashing, but I also observed it can take several minutes.
After a while you should get an output like this:
Analyzing file:///home/hayden/bug_report/./test_bad.mkv
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
g_mutex_clear() called on uninitialised or locked mutex
Aborted (core dumped)
A few findings and notes:
- The amount of frames in the video seems to have an impact - changing 13 to 16 seemingly makes it no longer reproducible. It is a bit weird given that the gst-discoverer probably shouldn’t decode all frames. I’d maybe expect it to try decoding one.
- The problem seems sensitive to timing - for example adding a verbose GST_DEBUG seems to make it much harder to reproduce.
- I was unable to reproduce this if the video was using a different codec. For example AV1 file generated like this did not cause a similar issue.
gst-launch-1.0 videotestsrc num_buffers=13 ! video/x-raw, framerate=15/1, width=1920, height=1080 ! nvvidconv ! nvv4l2av1enc ! matroskamux ! filesink location=./test_bad.mkv
- I am attaching:
- the
nvidia-bug-report-tegra.log
generated usingnvidia-bug-report-tegra.sh
- video file I generated (just in case freshly generated video won’t trigger it)
- gst-discoverer internal pipeline
.dot
graph
- the
- There are a few topics that also exhibited similar problems but those are old and were about deprecated
omx
plugins: - after catching it in a debugger I extracted this backtrace:
backtrace:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000fffff7a7caac in __GI_abort () at abort.c:79
#2 0x0000fffff7c6ea04 in g_mutex_clear () at /lib/aarch64-linux-gnu/libglib-2.0.so.0
#3 0x0000fffff7daba88 in gst_object_finalize (object=0xffffec212170) at gstobject.c:427
#4 0x0000fffff7d237e0 in g_object_unref () at /lib/aarch64-linux-gnu/libgobject-2.0.so.0
#5 0x0000fffff7e06134 in gst_pad_stop_task (pad=0xffffe8024ad0 [GstPad|src]) at gstpad.c:6312
#6 0x0000fffff547376c in () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so
#7 0x0000fffff7dde9bc in gst_element_change_state (element=element@entry=0xffffec13ab80 [GstElement|nvv4l2decoder0], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#8 0x0000fffff7ddf110 in gst_element_set_state_func (element=0xffffec13ab80 [GstElement|nvv4l2decoder0], state=<optimized out>) at gstelement.c:2919
#9 0x0000fffff7db8154 in gst_bin_element_set_state
(next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xffffec13ab80 [GstElement|nvv4l2decoder0], bin=0xaaaaaac6a2d0 [GstBin|decodebin1]) at gstbin.c:2605
#10 gst_bin_change_state_func (element=0xaaaaaac6a2d0 [GstElement|decodebin1], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#11 0x0000fffff6ef0b58 in () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstplayback.so
#12 0x0000fffff7dde9bc in gst_element_change_state (element=element@entry=0xaaaaaac6a2d0 [GstElement|decodebin1], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#13 0x0000fffff7ddf110 in gst_element_set_state_func (element=0xaaaaaac6a2d0 [GstElement|decodebin1], state=<optimized out>) at gstelement.c:2919
#14 0x0000fffff7db8154 in gst_bin_element_set_state
(next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xaaaaaac6a2d0 [GstElement|decodebin1], bin=0xaaaaaac640c0 [GstBin|discoverer-uri]) at gstbin.c:2605
#15 gst_bin_change_state_func (element=0xaaaaaac640c0 [GstElement|discoverer-uri], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#16 0x0000fffff6f07bd0 in () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstplayback.so
#17 0x0000fffff7dde9bc in gst_element_change_state (element=element@entry=0xaaaaaac640c0 [GstElement|discoverer-uri], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#18 0x0000fffff7ddf110 in gst_element_set_state_func (element=0xaaaaaac640c0 [GstElement|discoverer-uri], state=<optimized out>) at gstelement.c:2919
#19 0x0000fffff7db8154 in gst_bin_element_set_state
(next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xaaaaaac640c0 [GstElement|discoverer-uri], bin=0xaaaaaac541e0 [GstBin|Discoverer]) at gstbin.c:2605
#20 gst_bin_change_state_func (element=0xaaaaaac541e0 [GstElement|Discoverer], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#21 0x0000fffff7dde9bc in gst_element_change_state (element=element@entry=0xaaaaaac541e0 [GstElement|Discoverer], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#22 0x0000fffff7ddf110 in gst_element_set_state_func (element=0xaaaaaac541e0 [GstElement|Discoverer], state=<optimized out>) at gstelement.c:2919
#23 0x0000fffff7f7ef88 in () at /lib/aarch64-linux-gnu/libgstpbutils-1.0.so.0
#24 0x0000fffff7f7fa34 in gst_discoverer_discover_uri () at /lib/aarch64-linux-gnu/libgstpbutils-1.0.so.0
#25 0x0000aaaaaaaa45e8 in ()
#26 0x0000aaaaaaaa3168 in ()
#27 0x0000fffff7a7ce10 in __libc_start_main (main=0xaaaaaaaa2f50, argc=2, argv=0xfffffffff3b8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>)
at ../csu/libc-start.c:308
#28 0x0000aaaaaaaa3344 in ()
(The libgstnvvideo4linux2.so
library is from nvidia-l4t-gstreamer 35.4.1-20230801124926
package.)
nvv4l2decoder_crash_bug_report_files.zip (850.2 KB)