Extremely unreliable HW encoder

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

I’ve been experimenting with HW-accelerated encoding pretty much non-stop ever since I got my Nano a couple weeks back.

While doing so, I noticed that sometimes, regardless of my settings or the input, the “gst-launch-1.0” process would sit idle and never finish.
I started experimenting, trying to find a pattern to this behavior, and while there are certain input files that’d ALWAYS hang, for others it’s kind of a hit or miss.
Generally speaking, if I add audio the chance of the process hanging goes up several times, and if I add audio conversion to the mix it’d hang 50% of the times, or more. Running the same command several times (in a loop or parallel) almost guarantees one of the processes would hang.

Now it’s time to get technical. I’ll try to provide every piece of information to help with recreating what I experience:

  • OS: Ubuntu 18.04 (Jetson official image)
  • Kernel version: 4.9.140-tegra
  • Gstreamer vesion: 1.14.1

For the target video I chose the good old Big Buck Bunny (http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4).

Here’s the command I use:

gst-launch-1.0 filesrc location=BigBuckBunny.mp4 ! \
    qtdemux name=demux demux.video_0 ! queue ! \
    h264parse ! nvv4l2decoder enable-max-performance=1 ! \
    nvvidconv interpolation-method=5 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12 ! \
    nvv4l2h265enc bitrate=1000000 peak-bitrate=1100000 EnableTwopassCBR=1 control-rate=0 preset-level=4 \
    maxperf-enable=1 iframeinterval=60 qp-range="25,35:21,28:-1,-1" ! h265parse ! queue ! \
    qtmux name=mux ! filesink location=jetson-BigBuckBunny.mp4 \
    demux.audio_0 ! queue ! avdec_aac ! audioconvert ! voaacenc bitrate=64000 ! aacparse ! queue ! mux.

Here’s the output of the above command:

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading sys.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Redistribute latency...
NvMMLiteOpen : Block : BlockType = 8
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8
NVMEDIA: H265 : Profile : 1
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.  <-- Terminaled by <C-c>
Interrupt: Stopping pipeline ...
Execution ended after 0:20:33.033821132  <-- Job sat idle
Setting pipeline to PAUSED ...
Setting pipeline to READY ...

And the mediainfo of the output file “jetson-BigBuckBunny.mp4”:

General
Complete name                            : jetson-BigBuckBunny.mp4
Format                                   : MPEG-4
Format profile                           : QuickTime
Codec ID                                 : qt   2005.03 (qt  )
File size                                : 31.8 MiB

Please note that I deliberately chose to present a complex example, as it’s more prone to failure.
I could recreate the problem without audio conversion and with no options set for the video encode.
For example, this command would also hang: (not necessarily on the first try)

gst-launch-1.0 filesrc location=BigBuckBunny.mp4 ! \
    qtdemux name=demux demux.video_0 ! queue ! \
    h264parse ! nvv4l2decoder ! \
    nvv4l2h265enc ! h265parse ! queue ! \
    qtmux name=mux ! filesink location=jetson-BigBuckBunny.mp4 \
    demux.audio_0 ! queue ! mux.

Investigating furthermore, I set the GST_DEBUG=3 global variable and saw the following warning:

0x7f8015a8a0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1483:gst_v4l2_buffer_pool_dqbuf:<nvv4l2h264enc0:pool:sink> V4L2 provided buffer has bytesused 0 which is too small to include data_offset 0
0x7f800034a0 FIXME              h265parse gsth265parse.c:1712:gst_h265_parse_parse_frame:<h265parse0> Implement timestamp/duration interpolation based on SEI message

It’s actually similar to the warning mentioned on this thread: https://devtalk.nvidia.com/default/topic/1052057/l4t-32-1-problems-switching-from-omxh264enc-to-nvv4l2h264enc/
However, it’s on a different platform (TX2), and on H264 I also get this FIXME warning. Moreover, those messages show up even on successful tries.

Other things I did trying to mitigate the problem:

  1. Use queue2 instead of queue (which surprisingly improved the situation).
  2. Use the now deprecated OMX decoder/encoder.
  3. Randomly increasing buffers (since the queue is obviously getting stuck somewhere...)

I’ll appreciate any help with solving this issue :)

Thank you.

2

Hi,
We will try to reproduce the issue. You may also try omxh264dec and omxh265enc.

3

Hey,

I’ve been playing also with omx (as mentioned in my post). It’s been the same hit-or-miss experience, unfortunately.

On a side note, I have this mpeg2 dvd which I’m also experimenting on, and I’ve never been able to get it to finish an encode (the furthest I got was like half-way through).
I was using both nvv4l2decoder and omxmpeg2videodec with no success :(

Hope to maybe get a hint where I could improve this to try and stop the pipe from hanging.

4

Hi,
omxh264dec and omxh265enc should work fine on r32.1. Please refer to
[url]https://devtalk.nvidia.com/default/topic/979908/jetson-tx1/gstreamer-transcoding-performance-issue/post/5033461/#5033461[/url]

5

@11wallace11

I would guess that the pipeline starves at some point if the audio and video streams are out of sync too much. I would try the following:

  • Use a multiqueue instead of a regular queue to keep audio and video streams in sync

  • Write a small python script to launch the pipeline and connect to the multiqueues “overrund” and “underrun” signals to check for starvation

6

Thanks for the hint! Will definitely try that out and report back!

7

Hi,

I’ll make sure to do more tests using omx codecs.

However, I’m a little bit confused since that’s what it says on the manual: “The gst-omx plugin is deprecated in Linux for Tegra (L4T) Release 32.1. Use gst-v4l2 plugin instead.”

8

Hi,
For unifying Jetson platforms and desktop GPUs, we are moving to v4l2 interface. r32.1 is the first release to have gst-v4l2 plugins and we have SQA tests passed. However, there are remain usecases not being tested/verified. Thanks for reporting issues and we will keep improving the stability. Since omx plugins are used/maintained for many releases, it should be fine to run them on r32.1.