Extremely unreliable HW encoder

I’ve been experimenting with HW-accelerated encoding pretty much non-stop ever since I got my Nano a couple weeks back.

While doing so, I noticed that sometimes, regardless of my settings or the input, the “gst-launch-1.0” process would sit idle and never finish.
I started experimenting, trying to find a pattern to this behavior, and while there are certain input files that’d ALWAYS hang, for others it’s kind of a hit or miss.
Generally speaking, if I add audio the chance of the process hanging goes up several times, and if I add audio conversion to the mix it’d hang 50% of the times, or more. Running the same command several times (in a loop or parallel) almost guarantees one of the processes would hang.

Now it’s time to get technical. I’ll try to provide every piece of information to help with recreating what I experience:

  • OS: Ubuntu 18.04 (Jetson official image)
  • Kernel version: 4.9.140-tegra
  • Gstreamer vesion: 1.14.1

For the target video I chose the good old Big Buck Bunny (http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4).

Here’s the command I use:

gst-launch-1.0 filesrc location=BigBuckBunny.mp4 ! \
    qtdemux name=demux demux.video_0 ! queue ! \
    h264parse ! nvv4l2decoder enable-max-performance=1 ! \
    nvvidconv interpolation-method=5 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12 ! \
    nvv4l2h265enc bitrate=1000000 peak-bitrate=1100000 EnableTwopassCBR=1 control-rate=0 preset-level=4 \
    maxperf-enable=1 iframeinterval=60 qp-range="25,35:21,28:-1,-1" ! h265parse ! queue ! \
    qtmux name=mux ! filesink location=jetson-BigBuckBunny.mp4 \
    demux.audio_0 ! queue ! avdec_aac ! audioconvert ! voaacenc bitrate=64000 ! aacparse ! queue ! mux.

Here’s the output of the above command:

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading sys.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Redistribute latency...
NvMMLiteOpen : Block : BlockType = 8
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8
NVMEDIA: H265 : Profile : 1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.  <-- Terminaled by <C-c>
Interrupt: Stopping pipeline ...
Execution ended after 0:20:33.033821132  <-- Job sat idle
Setting pipeline to PAUSED ...
Setting pipeline to READY ...

And the mediainfo of the output file “jetson-BigBuckBunny.mp4”:

General
Complete name                            : jetson-BigBuckBunny.mp4
Format                                   : MPEG-4
Format profile                           : QuickTime
Codec ID                                 : qt   2005.03 (qt  )
File size                                : 31.8 MiB

Please note that I deliberately chose to present a complex example, as it’s more prone to failure.
I could recreate the problem without audio conversion and with no options set for the video encode.
For example, this command would also hang: (not necessarily on the first try)

gst-launch-1.0 filesrc location=BigBuckBunny.mp4 ! \
    qtdemux name=demux demux.video_0 ! queue ! \
    h264parse ! nvv4l2decoder ! \
    nvv4l2h265enc ! h265parse ! queue ! \
    qtmux name=mux ! filesink location=jetson-BigBuckBunny.mp4 \
    demux.audio_0 ! queue ! mux.

Investigating furthermore, I set the GST_DEBUG=3 global variable and saw the following warning:

0x7f8015a8a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1483:gst_v4l2_buffer_pool_dqbuf:<nvv4l2h264enc0:pool:sink> V4L2 provided buffer has bytesused 0 which is too small to include data_offset 0
0x7f800034a0 FIXME              h265parse gsth265parse.c:1712:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message

It’s actually similar to the warning mentioned on this thread: https://devtalk.nvidia.com/default/topic/1052057/l4t-32-1-problems-switching-from-omxh264enc-to-nvv4l2h264enc/
However, it’s on a different platform (TX2), and on H264 I also get this FIXME warning. Moreover, those messages show up even on successful tries.

Other things I did trying to mitigate the problem:

  1. Use queue2 instead of queue (which surprisingly improved the situation).
  2. Use the now deprecated OMX decoder/encoder.
  3. Randomly increasing buffers (since the queue is obviously getting stuck somewhere...)

I’ll appreciate any help with solving this issue :)

Thank you.

Hi,
We will try to reproduce the issue. You may also try omxh264dec and omxh265enc.

Hey,

I’ve been playing also with omx (as mentioned in my post). It’s been the same hit-or-miss experience, unfortunately.

On a side note, I have this mpeg2 dvd which I’m also experimenting on, and I’ve never been able to get it to finish an encode (the furthest I got was like half-way through).
I was using both nvv4l2decoder and omxmpeg2videodec with no success :(

Hope to maybe get a hint where I could improve this to try and stop the pipe from hanging.

Hi,
omxh264dec and omxh265enc should work fine on r32.1. Please refer to
https://devtalk.nvidia.com/default/topic/979908/jetson-tx1/gstreamer-transcoding-performance-issue/post/5033461/#5033461

@11wallace11

I would guess that the pipeline starves at some point if the audio and video streams are out of sync too much. I would try the following:

  • Use a multiqueue instead of a regular queue to keep audio and video streams in sync

  • Write a small python script to launch the pipeline and connect to the multiqueues “overrund” and “underrun” signals to check for starvation

Thanks for the hint! Will definitely try that out and report back!

Hi,

I’ll make sure to do more tests using omx codecs.

However, I’m a little bit confused since that’s what it says on the manual: “The gst-omx plugin is deprecated in Linux for Tegra (L4T) Release 32.1. Use gst-v4l2 plugin instead.”

Hi,
For unifying Jetson platforms and desktop GPUs, we are moving to v4l2 interface. r32.1 is the first release to have gst-v4l2 plugins and we have SQA tests passed. However, there are remain usecases not being tested/verified. Thanks for reporting issues and we will keep improving the stability. Since omx plugins are used/maintained for many releases, it should be fine to run them on r32.1.