Simple pipeline with nvv4l2decoder got stuck

denisvmedyantsev · June 25, 2021, 7:01am

Platform

NVIDIA GeForce RTX 2080
Driver Version: 465.19.01
Docker nvcr.io/nvidia/deepstream:5.0-20.07-base

> gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

but it’s okey with avdec/vaapidecode

> gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! avdec_h264 ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.220984700
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

GST_DEBUG=5 gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

...
0:00:00.614602755 19768 0x557836592cf0 DEBUG             GST_MEMORY gstmemory.c:139:gst_memory_init: new memory 0x5578365c8470, maxsize:71 offset:0 size:64
0:00:00.614610302 19768 0x557836592cf0 DEBUG        GST_PERFORMANCE gstallocator.c:465:_sysmem_copy: memcpy 64 memory 0x5578365c8390 -> 0x5578365c8470
0:00:00.614618503 19768 0x557836592cf0 DEBUG             GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f0d6000daf8 (GstVideoMeta) of size 112
0:00:00.614625121 19768 0x557836592cf0 DEBUG              videometa gstvideometa.c:86:gst_video_meta_transform: copy video metadata
0:00:00.614631365 19768 0x557836592cf0 DEBUG             GST_MEMORY gstmemory.c:88:_gst_memory_free: free memory 0x5578365c8390
0:00:00.614640158 19768 0x557836592cf0 DEBUG               GST_PADS gstpad.c:4072:gst_pad_query:<fakesink0:sink> sent query 0x7f0d6054f590 (allocation), result 0
0:00:00.614646925 19768 0x557836592cf0 DEBUG               GST_PADS gstpad.c:4117:gst_pad_query:<fakesink0:sink> query failed
0:00:00.614653488 19768 0x557836592cf0 DEBUG               GST_PADS gstpad.c:4238:gst_pad_peer_query:<nvv4l2decoder0:src> query failed
0:00:00.614660986 19768 0x557836592cf0 DEBUG                   v4l2 gstv4l2object.c:4389:gst_v4l2_object_stop:<nvv4l2decoder0:src> stopping

Video: output.mp4

Fiona.Chen · June 25, 2021, 8:26am

Can you upgraded to latest DeepStream SDK version 5.1? There are lots of bug fix with latest deepstream.
Can you upload your test mp4 file?

denisvmedyantsev · June 25, 2021, 8:34am

the same
docker run -it --rm --gpus=all -v /mnt/data:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base

root@640c831f53a9:/opt/nvidia/deepstream/deepstream-5.1# gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

(gst-plugin-scanner:13): GStreamer-WARNING **: 08:32:31.235: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

Can you upload your test mp4 file?

https://drive.google.com/file/d/1fYxEVz55uNWl0wcDY2M1uqtvV0W9C2VY/

pwoolvett · June 25, 2021, 11:23pm

Something similar happened to me a while ago…

What I did would be similar to the following:

$ docker run -it --rm --gpus=all --entrypoint gst-launch-1.0 -v /mnt/data/:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base -v -e filesrc location=/big-data/video/output.mp4 ! qtdemux ! avdec_h264 ! fpsdisplaysink video-sink=fakesink fps-update-interval=1

[...]

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 233, dropped: 0, current: 25.07, average: 23.40
Got EOS from element "pipeline0".
Execution ended after 0:00:10.010020069
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Comparing it with:

$ docker run -it --rm --gpus=all --entrypoint gst-launch-1.0 -v /mnt/data/:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base -v -e filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fpsdisplaysink video-sink=fakesink  fps-update-interval=1

[...]

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 74, dropped: 0, current: 7.55, average: 25.14
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 75, dropped: 0, current: 16.25, average: 24.96
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.1, profile=(string)main, width=(int)1280, height=(int)720, framerate=(fraction)100000/4273, pixel-aspect-ratio=(fraction)1/1, multiview-mode=(string)row-interleaved, multiview-flags=(GstVideoMultiviewFlagsSet)4000:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped+half-aspect/mixed-mono, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 76, dropped: 0, current: 556.35, average: 25.27
/GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.1, profile=(string)main, width=(int)1280, height=(int)720, framerate=(fraction)100000/4273, pixel-aspect-ratio=(fraction)1/1, multiview-mode=(string)row-interleaved, multiview-flags=(GstVideoMultiviewFlagsSet)4000:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped+half-aspect/mixed-mono, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true
/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)3.1, profile=(string)main, width=(int)1280, height=(int)720, framerate=(fraction)100000/4273, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true

I think there might be something weird around frame number 76 (?), where this the caps changed from multiview-mode=(string)mono to multiview-mode=(string)row-interleaved.

I have absolutely no idea what I’m talking about, but decoding and re-encoding the video again solved the issue in my case,

$ docker run -it --rm --gpus=all --entrypoint gst-launch-1.0 -v /mnt/data/:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base -v -e  filesrc location=/big-data/video/output.mp4 ! qtdemux ! avdec_h264 ! x264enc ! qtmux ! filesink location=/big-data/video/output_re-encoded.mp4

and then it works fine,

$ docker run -it --rm --gpus=all --entrypoint gst-launch-1.0 -v /mnt/data/:/big-data nvcr.io/nvidia/deepstream:5.1-21.02-base -v -e filesrc location=/big-data/video/output_re-encoded.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink

kudryavtsev_ia · June 26, 2021, 2:46am

There are workarounds like that, but they are not solution. Speaking about real-time streamed processing, the thing deepstream was developed, right now the camera can attack the pipeline by injecting the bad frame, causing it to hang. It’s absolutely unacceptable in production-grade systems and must be fixed. Even if it crash instead of stuck it would work better, but ideally it must skip the frame like cpu-based decoders do.

We have a system where the bug was found (initial post posted by our engineer), our 3rd-party engineering partner has confirmed it in a separate hardware/software stack. So, practically, DeepStream is vulnerable to streams with specially injected data.

NVDEC/NVENC is the thing which makes DeepStream commercially efficient, switching to cpu decoding/encoding ruines resource balance and causing overall performance fall because of memory copying for full-scale raw images from CPU to GPU.

hardik.panchal · August 5, 2022, 6:06am

I am also facing a similar kind of issue with the nvv4l2decoder plugin. Seems like h/w decoder plugin from nvidia is not able to process some frames and got into a hang state.

The issue here is we are not able to recover from this state in our application using any kind of events such as EOS, flushing, or anything.

When I tried iterating over child plugins in decodebin, and setting them to NULL, it hangs at nvv4l2decoder.
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:292 decodebin3-3 found
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:292 nvv4l2decoder3 found
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:303 setting nvv4l2decoder3 to null

Fiona.Chen · August 12, 2022, 6:46am

The bug is fixed internally. Please wait for the next release.

system · August 26, 2022, 6:46am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

yingliu · September 6, 2022, 7:24am

Fixed in DeepStream 6.1.1.