What is an ffmpeg command line that can be used so the final mp4 can always be played on a Jetson AGX hardware?

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Our application is used to playback MP4 files using the NVDEC accelerated device on a Jetson AGX.

That part works great with about 50% of our MP4 videos. The other 50%, the device generates an error saying the encoding is not supported.

We have two types of videos, some that are created from one image and others that are actual 4K videos.

Here is the command that I use to create a video from one image:

ffmpeg -hide_banner -y -loop 1 -i "input-image.jpg" -c:v libx264 -t 1 -vf format=yuv420p -vf scale=3840:2160 -filter:v fps=fps=30 -movflags +faststart -an "out.mp4"

And here is the command I use to create videos from frames:

ffmpeg -hide_banner -y -i "f%d.jpg" -c:v libx264 -vf format=yuv420p -filter:v fps=fps=30 -movflags +faststart -an "out.mp4"

As I mentioned above, these work only about 50% of the time. So I would imagine that the ffmpeg library makes a decision on some of the JPEG data which are incompatible with the NVDEC decoder. What I would need are the command line options that will make sure the encoding works every single time.

Note that I run these commands on a cloud computer, normal Intel based processor with the default ffmpeg from Ubuntu 18.04. I do not want to do the encoding on the Jetson. The cloud computer will host the videos so I want that to remain there. I know I could otherwise use a Jetson card NVENC to make it work (propbably), but that is not practical in our current environment.

Thank you.

3

Hi,
It looks like you decode JPEGs into YUVs and then encode into h264 with software jpeg decoder and h264 encoder. Seems to be an issue in ffmpeg. What is the version of ffmpeg?

4

Here is the version info:

$ ffmpeg -version
ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
libavutil      55. 78.100 / 55. 78.100
libavcodec     57.107.100 / 57.107.100
libavformat    57. 83.100 / 57. 83.100
libavdevice    57. 10.100 / 57. 10.100
libavfilter     6.107.100 /  6.107.100
libavresample   3.  7.  0 /  3.  7.  0
libswscale      4.  8.100 /  4.  8.100
libswresample   2.  9.100 /  2.  9.100
libpostproc    54.  7.100 / 54.  7.100

I have the ffmpeg stock version on Ubuntu 18.04 on standard Intel machines. So you should be able to reproduce the issue on any standard computer.

If you need, I have a picture I can share one which generates the problem.

5

Anything I can do here?

Would you have a document that I’ve missed that gives us ffmpeg command lines that work every time?

6

Hi,
We have a package to enable hardware decoding but your pipeline uses software encoding, it should be native ffmpeg code. We will try to replicate the issue.

1 Like
7

Hi,
We try the commands on r32.5/Xavier and do not see any issue:

nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/09_camera_jpeg_capture$ ./camera_jpeg_capture --cap-time 5                                                [INFO] (NvEglRenderer.cpp:110) <renderer0> Setting Screen width 640 height 480
PRODUCER: Creating output stream
PRODUCER: Launching consumer thread
CONSUMER: Waiting until producer is connected...
CONSUMER: Waiting until producer is connected...
PRODUCER: Available Sensor modes :
PRODUCER: [0] W=2592 H=1944
PRODUCER: [1] W=2592 H=1458
PRODUCER: [2] W=1280 H=720
PRODUCER: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
CONSUMER: Producer has connected; continuing.
CONSUMER: Done.
CONSUMER: Done.
PRODUCER: Done -- exiting.
nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/09_camera_jpeg_capture$ ffmpeg -hide_banner -y -i "output%03d.jpg" -c:v libx264 -vf format=yuv420p -filter:v fps=fps=30 -movflags +faststart -an "out.mp4"
Input #0, image2, from 'output%03d.jpg':
  Duration: 00:00:06.00, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 1920x1080, 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x55671ada00] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x55671ada00] profile High, level 4.0
[libx264 @ 0x55671ada00] 264 - core 152 r2854 e9a5903 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuvj420p(pc), 1920x1080, q=-1--1, 30 fps, 15360 tbn, 30 tbc
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=   42 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   47 fps= 30 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   frame=   55 fps= 26 q=29.0 size=     256kB time=00:00:00.06 bitrate=31432.2kbitsframe=   59 fps= 22 q=29.0 size=     512kB time=00:00:00.20 bitrate=20966.6kbitsframe=   65 fps= 19 q=29.0 size=     768kB time=00:00:00.40 bitrate=15727.0kbitsframe=   71 fps= 17 q=29.0 size=    1280kB time=00:00:00.60 bitrate=17475.0kbitsframe=   77 fps= 16 q=29.0 size=    1792kB time=00:00:00.80 bitrate=18349.1kbitsframe=   84 fps= 15 q=29.0 size=    2048kB time=00:00:01.03 bitrate=16235.4kbitsframe=   90 fps= 14 q=29.0 size=    2560kB time=00:00:01.23 bitrate=17003.4kbitsframe=   97 fps= 14 q=29.0 size=    3072kB time=00:00:01.46 bitrate=17158.0kbitsframe=  102 fps= 13 q=29.0 size=    3328kB time=00:00:01.63 bitrate=16691.2kbitsframe=  107 fps= 13 q=29.0 size=    3584kB time=00:00:01.80 bitrate=16310.8kbitsframe=  113 fps= 13 q=29.0 size=    4096kB time=00:00:02.00 bitrate=16776.9kbitsframe=  119 fps= 13 q=29.0 size=    4608kB time=00:00:02.20 bitrate=17158.2kbitsframe=  121 fps= 12 q=29.0 size=    4608kB time=00:00:02.26 bitrate=16653.5kbitsframe=  130 fps= 12 q=29.0 size=    5376kB time=00:00:02.56 bitrate=17158.2kbitsframe=  137 fps= 12 q=29.0 size=    5888kB time=00:00:02.80 bitrate=17226.3kbitsframe=  143 fps= 12 q=29.0 size=    6144kB time=00:00:03.00 bitrate=16777.0kbitsframe=  149 fps= 11 q=29.0 size=    6656kB time=00:00:03.20 bitrate=17039.1kbitsframe=  155 fps= 11 q=29.0 size=    6912kB time=00:00:03.40 bitrate=16653.6kbitsframe=  161 fps= 11 q=29.0 size=    7424kB time=00:00:03.60 bitrate=16893.5kbitsframe=  164 fps= 11 q=29.0 size=    7680kB time=00:00:03.70 bitrate=17003.7kbitsframe=  168 fps= 11 q=29.0 size=    7936kB time=00:00:03.83 bitrate=16959.4kbitsframe=  175 fps= 11 q=29.0 size=    8448kB time=00:00:04.06 bitrate=17017.7kbitsframe=  179 fps= 11 q=29.0 size=    8704kB time=00:00:04.20 bitrate=16976.8kbits[mp4 @ 0x55671abe90] Starting second pass: moving the moov atom to the beginning of the file
frame=  180 fps=8.2 q=-1.0 Lsize=   12419kB time=00:00:05.90 bitrate=17242.9kbits/s speed=0.268x
video:12416kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.024643%
[libx264 @ 0x55671ada00] frame I:2     Avg QP:22.68  size:150047
[libx264 @ 0x55671ada00] frame P:49    Avg QP:25.16  size:114245
[libx264 @ 0x55671ada00] frame B:129   Avg QP:25.98  size: 52828
[libx264 @ 0x55671ada00] consecutive B-frames:  3.3%  3.3%  0.0% 93.3%
[libx264 @ 0x55671ada00] mb I  I16..4:  0.9% 98.8%  0.3%
[libx264 @ 0x55671ada00] mb P  I16..4:  0.1% 56.6%  0.0%  P16..4: 20.5%  8.9%  6.3%  0.0%  0.0%    skip: 7.6%
[libx264 @ 0x55671ada00] mb B  I16..4:  0.1% 16.3%  0.0%  B16..8: 21.1%  6.6%  2.5%  direct: 6.9%  skip:46.5%  L0:51.0% L1:33.4% BI:15.5%
[libx264 @ 0x55671ada00] 8x8 transform intra:99.5% inter:95.5%
[libx264 @ 0x55671ada00] coded y,uvDC,uvAC intra: 93.4% 33.0% 0.3% inter: 31.3% 15.4% 0.1%
[libx264 @ 0x55671ada00] i16 v,h,dc,p: 29% 66%  5%  1%
[libx264 @ 0x55671ada00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  5%  4% 82%  2%  1%  1%  1%  2%  2%
[libx264 @ 0x55671ada00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 27% 24% 19%  2%  1%  1%  6% 16%  4%
[libx264 @ 0x55671ada00] i8c dc,h,v,p: 60% 22% 18%  0%
[libx264 @ 0x55671ada00] Weighted P-Frames: Y:12.2% UV:12.2%
[libx264 @ 0x55671ada00] ref P L0: 34.6%  7.8% 39.1% 18.3%  0.3%
[libx264 @ 0x55671ada00] ref B L0: 59.9% 29.3% 10.8%
[libx264 @ 0x55671ada00] ref B L1: 86.0% 14.0%
[libx264 @ 0x55671ada00] kb/s:16950.63
nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/09_camera_jpeg_capture$ gst-launch-1.0 filesrc location= out.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvoverlaysink
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:06.000028579
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Please check if you can run the commands successfully. Not sure but probably it is specific to the JPEG files.

8

Okay, I found the problem. Strange, though but I have to force the pixel format to yuv420p otherwise it may remain at a higher scale such as yuv444p or yuv422p and the NVDEC does not support those.

I thought that the -vf format=yuv420p was sufficient, but apparently not. What I had to add is the -pix_fmt yuv420p command line option to make it work properly. Now the NVDEC is happy whatever the original JPEG input data.

If you’d like a good YUV 444 image that will give you grief, here is one on Wikipedia.

The scale of the default command I have above will be ignored and the output movie will fail.

When you try to add the -pix_fmt yuv420p, it also fails because the width is not a multiple of 2. It would be great, though, if we could get error messages that tell us what’s wrong. What I see now is:

NVMEDIA: NVMEDIABufferProcessing: 1507: NvMediaParserParse Unsupported Codec
Event_BlockError from 0BlockH264Dec : Error code - e3840
Sending error event from 0BlockH253DecNVMEDIA: NnMMLitemNVMEDIADecDoWork: 1985: NVMEDIA Video Dec Unsupported Stream

So it tells us it’s an unsupported Codec, but it is really not clear since most of us expect all H.264 encoded movies to work when only yuv420p is supported.

1 Like