Simple pipeline with nvv4l2decoder got stuck

Platform

Simple pipeline with nvv4l2decoder got stuck

> gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

but it’s okey with avdec/vaapidecode

> gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! avdec_h264 ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.220984700
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

GST_DEBUG=5 gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

...
0:00:00.614602755 19768 0x557836592cf0 DEBUG             GST_MEMORY gstmemory.c:139:gst_memory_init: new memory 0x5578365c8470, maxsize:71 offset:0 size:64
0:00:00.614610302 19768 0x557836592cf0 DEBUG        GST_PERFORMANCE gstallocator.c:465:_sysmem_copy: memcpy 64 memory 0x5578365c8390 -> 0x5578365c8470
0:00:00.614618503 19768 0x557836592cf0 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f0d6000daf8 (GstVideoMeta) of size 112
0:00:00.614625121 19768 0x557836592cf0 DEBUG              videometa gstvideometa.c:86:gst_video_meta_transform: copy video metadata
0:00:00.614631365 19768 0x557836592cf0 DEBUG             GST_MEMORY gstmemory.c:88:_gst_memory_free: free memory 0x5578365c8390
0:00:00.614640158 19768 0x557836592cf0 DEBUG               GST_PADS gstpad.c:4072:gst_pad_query:<fakesink0:sink> sent query 0x7f0d6054f590 (allocation), result 0
0:00:00.614646925 19768 0x557836592cf0 DEBUG               GST_PADS gstpad.c:4117:gst_pad_query:<fakesink0:sink> query failed
0:00:00.614653488 19768 0x557836592cf0 DEBUG               GST_PADS gstpad.c:4238:gst_pad_peer_query:<nvv4l2decoder0:src> query failed
0:00:00.614660986 19768 0x557836592cf0 DEBUG                   v4l2 gstv4l2object.c:4389:gst_v4l2_object_stop:<nvv4l2decoder0:src> stopping

Video: output.mp4

Can you upgraded to latest DeepStream SDK version 5.1? There are lots of bug fix with latest deepstream.
Can you upload your test mp4 file?

the same
docker run -it --rm --gpus=all -v /mnt/data:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base

root@640c831f53a9:/opt/nvidia/deepstream/deepstream-5.1# gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

(gst-plugin-scanner:13): GStreamer-WARNING **: 08:32:31.235: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

Can you upload your test mp4 file?

Something similar happened to me a while ago…

What I did would be similar to the following:

$ docker run -it --rm --gpus=all --entrypoint gst-launch-1.0 -v /mnt/data/:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base -v -e filesrc location=/big-data/video/output.mp4 ! qtdemux ! avdec_h264 ! fpsdisplaysink video-sink=fakesink fps-update-interval=1

[...]

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 233, dropped: 0, current: 25.07, average: 23.40
Got EOS from element "pipeline0".
Execution ended after 0:00:10.010020069
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Comparing it with:

$ docker run -it --rm --gpus=all --entrypoint gst-launch-1.0 -v /mnt/data/:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base -v -e filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fpsdisplaysink video-sink=fakesink  fps-update-interval=1

[...]

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 74, dropped: 0, current: 7.55, average: 25.14
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 75, dropped: 0, current: 16.25, average: 24.96
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.1, profile=(string)main, width=(int)1280, height=(int)720, framerate=(fraction)100000/4273, pixel-aspect-ratio=(fraction)1/1, multiview-mode=(string)row-interleaved, multiview-flags=(GstVideoMultiviewFlagsSet)4000:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped+half-aspect/mixed-mono, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 76, dropped: 0, current: 556.35, average: 25.27
/GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.1, profile=(string)main, width=(int)1280, height=(int)720, framerate=(fraction)100000/4273, pixel-aspect-ratio=(fraction)1/1, multiview-mode=(string)row-interleaved, multiview-flags=(GstVideoMultiviewFlagsSet)4000:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped+half-aspect/mixed-mono, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.1, profile=(string)main, width=(int)1280, height=(int)720, framerate=(fraction)100000/4273, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true

I think there might be something weird around frame number 76 (?), where this the caps changed from multiview-mode=(string)mono to multiview-mode=(string)row-interleaved.

I have absolutely no idea what I’m talking about, but decoding and re-encoding the video again solved the issue in my case,

$ docker run -it --rm --gpus=all --entrypoint gst-launch-1.0 -v /mnt/data/:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base -v -e  filesrc location=/big-data/video/output.mp4 ! qtdemux ! avdec_h264 ! x264enc ! qtmux ! filesink location=/big-data/video/output_re-encoded.mp4

and then it works fine,

$ docker run -it --rm --gpus=all --entrypoint gst-launch-1.0 -v /mnt/data/:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base -v -e filesrc location=/big-data/video/output_re-encoded.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

There are workarounds like that, but they are not solution. Speaking about real-time streamed processing, the thing deepstream was developed, right now the camera can attack the pipeline by injecting the bad frame, causing it to hang. It’s absolutely unacceptable in production-grade systems and must be fixed. Even if it crash instead of stuck it would work better, but ideally it must skip the frame like cpu-based decoders do.

We have a system where the bug was found (initial post posted by our engineer), our 3rd-party engineering partner has confirmed it in a separate hardware/software stack. So, practically, DeepStream is vulnerable to streams with specially injected data.

NVDEC/NVENC is the thing which makes DeepStream commercially efficient, switching to cpu decoding/encoding ruines resource balance and causing overall performance fall because of memory copying for full-scale raw images from CPU to GPU.

1 Like