Nvv4l2decoder crashing in GstDiscoverer on h264 videos

We are observing an abort coming from within nvv4l2decoder/libgstnvvideo4linux2.so when using gst discoverer on an h264 encoded file.

Reproduction script tested on 2 jetson boxes:

  • Orin NX: # R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug 1 19:57:35 UTC 2023
  • Xavier NX: # R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020
#prepare bad test file
gst-launch-1.0 videotestsrc num_buffers=13 ! video/x-raw, framerate=15/1, width=1920, height=1080 ! nvvidconv ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=./test_bad.mkv

# try in a loop until a crash
while true
do
    gst-discoverer-1.0 ./test_bad.mkv || break
done

Sometimes the loop will run for a few seconds before crashing, but I also observed it can take several minutes.

After a while you should get an output like this:

Analyzing file:///home/hayden/bug_report/./test_bad.mkv
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
g_mutex_clear() called on uninitialised or locked mutex
Aborted (core dumped)

A few findings and notes:

  • The amount of frames in the video seems to have an impact - changing 13 to 16 seemingly makes it no longer reproducible. It is a bit weird given that the gst-discoverer probably shouldn’t decode all frames. I’d maybe expect it to try decoding one.
  • The problem seems sensitive to timing - for example adding a verbose GST_DEBUG seems to make it much harder to reproduce.
  • I was unable to reproduce this if the video was using a different codec. For example AV1 file generated like this did not cause a similar issue. gst-launch-1.0 videotestsrc num_buffers=13 ! video/x-raw, framerate=15/1, width=1920, height=1080 ! nvvidconv ! nvv4l2av1enc ! matroskamux ! filesink location=./test_bad.mkv
  • I am attaching:
    • the nvidia-bug-report-tegra.log generated using nvidia-bug-report-tegra.sh
    • video file I generated (just in case freshly generated video won’t trigger it)
    • gst-discoverer internal pipeline .dot graph
  • There are a few topics that also exhibited similar problems but those are old and were about deprecated omx plugins:
  • after catching it in a debugger I extracted this backtrace:
backtrace:
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000fffff7a7caac in __GI_abort () at abort.c:79
#2  0x0000fffff7c6ea04 in g_mutex_clear () at /lib/aarch64-linux-gnu/libglib-2.0.so.0
#3  0x0000fffff7daba88 in gst_object_finalize (object=0xffffec212170) at gstobject.c:427
#4  0x0000fffff7d237e0 in g_object_unref () at /lib/aarch64-linux-gnu/libgobject-2.0.so.0
#5  0x0000fffff7e06134 in gst_pad_stop_task (pad=0xffffe8024ad0 [GstPad|src]) at gstpad.c:6312
#6  0x0000fffff547376c in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so
#7  0x0000fffff7dde9bc in gst_element_change_state (element=element@entry=0xffffec13ab80 [GstElement|nvv4l2decoder0], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#8  0x0000fffff7ddf110 in gst_element_set_state_func (element=0xffffec13ab80 [GstElement|nvv4l2decoder0], state=<optimized out>) at gstelement.c:2919
#9  0x0000fffff7db8154 in gst_bin_element_set_state
    (next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xffffec13ab80 [GstElement|nvv4l2decoder0], bin=0xaaaaaac6a2d0 [GstBin|decodebin1]) at gstbin.c:2605
#10 gst_bin_change_state_func (element=0xaaaaaac6a2d0 [GstElement|decodebin1], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#11 0x0000fffff6ef0b58 in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstplayback.so
#12 0x0000fffff7dde9bc in gst_element_change_state (element=element@entry=0xaaaaaac6a2d0 [GstElement|decodebin1], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#13 0x0000fffff7ddf110 in gst_element_set_state_func (element=0xaaaaaac6a2d0 [GstElement|decodebin1], state=<optimized out>) at gstelement.c:2919
#14 0x0000fffff7db8154 in gst_bin_element_set_state
    (next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xaaaaaac6a2d0 [GstElement|decodebin1], bin=0xaaaaaac640c0 [GstBin|discoverer-uri]) at gstbin.c:2605
#15 gst_bin_change_state_func (element=0xaaaaaac640c0 [GstElement|discoverer-uri], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#16 0x0000fffff6f07bd0 in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstplayback.so
#17 0x0000fffff7dde9bc in gst_element_change_state (element=element@entry=0xaaaaaac640c0 [GstElement|discoverer-uri], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#18 0x0000fffff7ddf110 in gst_element_set_state_func (element=0xaaaaaac640c0 [GstElement|discoverer-uri], state=<optimized out>) at gstelement.c:2919
#19 0x0000fffff7db8154 in gst_bin_element_set_state
    (next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xaaaaaac640c0 [GstElement|discoverer-uri], bin=0xaaaaaac541e0 [GstBin|Discoverer]) at gstbin.c:2605
#20 gst_bin_change_state_func (element=0xaaaaaac541e0 [GstElement|Discoverer], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#21 0x0000fffff7dde9bc in gst_element_change_state (element=element@entry=0xaaaaaac541e0 [GstElement|Discoverer], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#22 0x0000fffff7ddf110 in gst_element_set_state_func (element=0xaaaaaac541e0 [GstElement|Discoverer], state=<optimized out>) at gstelement.c:2919
#23 0x0000fffff7f7ef88 in  () at /lib/aarch64-linux-gnu/libgstpbutils-1.0.so.0
#24 0x0000fffff7f7fa34 in gst_discoverer_discover_uri () at /lib/aarch64-linux-gnu/libgstpbutils-1.0.so.0
#25 0x0000aaaaaaaa45e8 in  ()
#26 0x0000aaaaaaaa3168 in  ()
#27 0x0000fffff7a7ce10 in __libc_start_main (main=0xaaaaaaaa2f50, argc=2, argv=0xfffffffff3b8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>)
    at ../csu/libc-start.c:308
#28 0x0000aaaaaaaa3344 in  ()

(The libgstnvvideo4linux2.so library is from nvidia-l4t-gstreamer 35.4.1-20230801124926 package.)

nvv4l2decoder_crash_bug_report_files.zip (850.2 KB)

1 Like

Hi,
The latest version for Orin series is Jetpack 6.1. Would be great if you can upgrade to the version and try. To check if the issue is still present on latest release.

@DaneLLL

Unfortunately I do not currently have capacity to attempt jetpack 6.1 at the moment.
But If you do have access to a 6.1 device then please follow the reproduce instruction I provided:

It’s just a few bash commands you can paste into the console.
The only possible caveat being the installation of the decoder and the discoverer: (gstreamer1.0-plugins-base-apps and nvidia-l4t-gstreamer packages)

Hi,
You are using Jetpack 5.1.2(r35.4.1). For checking further on the version, you may download source code of nvv4l2decoder:

Jetson Linux 35.4.1 | NVIDIA Developer
Driver Package (BSP) Sources

And manually build the plugin to debug further.

We will check on Jetpack 6.1.

I did try but I am unable to find the missing gstnvdsseimeta.h header.
dpkg -S doesn’t see it in any packages, It seems to be a part of deepstream SDK, but after downloading and checking the contents of jetson deepstream 6.2, 6.3 and 7.1: none of them had this header.

I did notice that they exist in the x86 version of deepstream SDK, not sure what’s the expected build workflow for jetson in this situation. I’ll probably try to frankenstein something to get it somewhat working but I expect issues.

If you could share what’s the expected build environment then that could expedite things.
Maybe it’s some crosscompiling-ready x86 docker for BSP customization, but I don’t immediately see it.

nevermind, it seems like there was no giant dependency tree on the header or lib side so just copying 2 headers from x86 SDK was enough to compile the plugin for jetson.

And for that 5.1.2 source you linked I managed to get this backtrace that showed gstv4l2videodec.c:2498
I’ll try to find some tome to analyze things further but it’ll probably take a while.

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000fffff7a78aac in __GI_abort () at abort.c:79
#2  0x0000fffff7c6aa04 in g_mutex_clear () at /lib/aarch64-linux-gnu/libglib-2.0.so.0
#3  0x0000fffff7da7a88 in gst_object_finalize (object=0xffffec211170) at gstobject.c:427
#4  0x0000fffff7d1f7e0 in g_object_unref () at /lib/aarch64-linux-gnu/libgobject-2.0.so.0
#5  0x0000fffff7e02134 in gst_pad_stop_task (pad=0xffffe8026ac0 [GstPad|src]) at gstpad.c:6312
#6  0x0000fffff5445d60 in gst_v4l2_video_dec_change_state (element=0xffffec13ab00 [GstElement|nvv4l2decoder0], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstv4l2videodec.c:2498
#7  0x0000fffff7dda9bc in gst_element_change_state (element=element@entry=0xffffec13ab00 [GstElement|nvv4l2decoder0], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#8  0x0000fffff7ddb110 in gst_element_set_state_func (element=0xffffec13ab00 [GstElement|nvv4l2decoder0], state=<optimized out>) at gstelement.c:2919
#9  0x0000fffff7db4154 in gst_bin_element_set_state (next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xffffec13ab00 [GstElement|nvv4l2decoder0], bin=0xaaaaaac6c2d0 [GstBin|decodebin1]) at gstbin.c:2605
#10 gst_bin_change_state_func (element=0xaaaaaac6c2d0 [GstElement|decodebin1], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#11 0x0000fffff6eecb58 in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstplayback.so
#12 0x0000fffff7dda9bc in gst_element_change_state (element=element@entry=0xaaaaaac6c2d0 [GstElement|decodebin1], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#13 0x0000fffff7ddb110 in gst_element_set_state_func (element=0xaaaaaac6c2d0 [GstElement|decodebin1], state=<optimized out>) at gstelement.c:2919
#14 0x0000fffff7db4154 in gst_bin_element_set_state (next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xaaaaaac6c2d0 [GstElement|decodebin1], bin=0xaaaaaac660c0 [GstBin|discoverer-uri]) at gstbin.c:2605
#15 gst_bin_change_state_func (element=0xaaaaaac660c0 [GstElement|discoverer-uri], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#16 0x0000fffff6f03bd0 in  () at /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstplayback.so
#17 0x0000fffff7dda9bc in gst_element_change_state (element=element@entry=0xaaaaaac660c0 [GstElement|discoverer-uri], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#18 0x0000fffff7ddb110 in gst_element_set_state_func (element=0xaaaaaac660c0 [GstElement|discoverer-uri], state=<optimized out>) at gstelement.c:2919
#19 0x0000fffff7db4154 in gst_bin_element_set_state (next=GST_STATE_READY, current=GST_STATE_PAUSED, start_time=0 [0:00:00.000000000], base_time=0 [0:00:00.000000000], element=0xaaaaaac660c0 [GstElement|discoverer-uri], bin=0xaaaaaac601e0 [GstBin|Discoverer]) at gstbin.c:2605
#20 gst_bin_change_state_func (element=0xaaaaaac601e0 [GstElement|Discoverer], transition=GST_STATE_CHANGE_PAUSED_TO_READY) at gstbin.c:2947
#21 0x0000fffff7dda9bc in gst_element_change_state (element=element@entry=0xaaaaaac601e0 [GstElement|Discoverer], transition=transition@entry=GST_STATE_CHANGE_PAUSED_TO_READY) at gstelement.c:2965
#22 0x0000fffff7ddb110 in gst_element_set_state_func (element=0xaaaaaac601e0 [GstElement|Discoverer], state=<optimized out>) at gstelement.c:2919
#23 0x0000fffff7f7af88 in  () at /lib/aarch64-linux-gnu/libgstpbutils-1.0.so.0
#24 0x0000fffff7f7ba34 in gst_discoverer_discover_uri () at /lib/aarch64-linux-gnu/libgstpbutils-1.0.so.0
#25 0x0000aaaaaaaa45e8 in  ()
#26 0x0000aaaaaaaa3168 in  ()
#27 0x0000fffff7a78e10 in __libc_start_main (main=0xaaaaaaaa2f50, argc=2, argv=0xfffffffff388, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#28 0x0000aaaaaaaa3344 in  ()