YUV image which we get from ffmpeg is wrong

Hello,

We took Nvidia’s ffmpeg version which has support for hardware-based video decoding on Jetson (Tegra chipset). Namely this one: https://github.com/jocover/jetson-ffmpeg. I compiled it and integrated it with our Virgo app. Compilation worked fine and the resulting executable runs without crashes on Jetsons with the L4T version:

head -n 1 /etc/nv_tegra_release

R32 (release), REVISION: 3.1, GCID: 18186506, BOARD: t186ref, EABI: aarch64, DATE: Tue Dec 10 07:03:07 UTC 2019

Note that it crashes on R31 release. But that’s okay because we detect the L4T version at runtime and enable the hardware acceleration on >= R32 platforms only.

So far so good. Now here is the problem we’re seeing:

The YUV image which we get out of this ffmpeg version is incorrectly sized / formatted if we enable the “h264_nvmpi” video decoder (this is the hardware decoder).

So if we do this:

// Force GPU accelerator decoder on Nvidia Jetson devices
        pCodec = avcodec_find_decoder_by_name("h264_nvmpi");
        pPlayer->formatContext->video_codec = pCodec;

…
if ((err = avformat_open_input(&pPlayer->formatContext, pPlayer->url, NULL, &pPlayer->options)) < 0) {
...

Then after video decoding and color conversion to RGB we get an image that looks like the linked image. So the image width is only half of what it should be and the colors are wrong. Note that it doesn’t matter how we do the color conversion. Whether we do it with ffmpeg’s own sws_scale() function or Nvidia’s own nppiYUV420ToBGR_8u_P3C4R() function. This suggests that the YUV image which we get from ffmpeg is already wrong. However the metadata in the AVFrame looks okay though: It’s YUV420P at a line size of 1920 (Y) and 960 (U and V). But that’s just the metadata.

Note that everything works as expected if we initialize ffmpeg with the software “h264” decoder. In that case the image has the correct size and colors using the same color conversion code that we use for the “h264_nvmpi” case.

Is there something special we need to do when we initialize the video decoder? Is this a known bug in the Tegra ffmpeg implementation?

https://imgur.com/Wz3PQ6T

Carl

Hi,
The github code is from community and you may contact the author to get some information. We are also developing ffmpeg with hard acceleration and it should be ready in next r32 release.

Do you have an ETA on that release

replace /usr/src/jetson_multimedia_api/include/nvbuf_utils.h to jetson-ffmpeg/include/nvbuf_utils.h file

different L4T Multimedia API version have different NvBufferColorFormat enums
on jetpack 4.2 the NvBufferColorFormat_YUV420_709 value is 23. but on jetpack 4.3 NvBufferColorFormat_YUV420_709 value is 27