Is it possible to use ffmpeg with h264_cuvid on the Jetson AGX Xavier?

I have Ubuntu 18.04 on a Desktop computer with an RTX 2060 and I can see that ffmpeg supports hardware acceleration like so:

$ ffmpeg -codecs 2>/dev/null | grep h264
 DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_crystalhd h264_v4l2m2m h264_vdpau h264_cuvid ) (encoders: libx264 libx264rgb h264_nvenc h264_omx h264_v4l2m2m h264_vaapi nvenc nvenc_h264 )

The h264_cuvid is what gets used to decompress my 4K video on my RTX board.

However, on the Jetson AGX Xaview, the ffmpeg command outputs:

 DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_v4l2m2m h264_vdpau ) (encoders: libx264 libx264rgb h264_omx h264_v4l2m2m h264_vaapi )

As we can see, no h264_cuvid

The same version of ffmpeg is installed on both machines. A ton of CUDA and NVidia packages (all?) are also installed (more than on my machine with the RTX 2060). So… how do I get ffmpeg to use hardware acceleration?

Note 1: I’ve been able to use the acceleration through gstreamer, but I can’t use that in my software for I need two videos as textures. So I really need to get the acceleration and it has to work with a library that gives me the necessary flexibility to use more than just a video output of a movie on the display.

Note 2: When I open the nvidia-settings tool on my Desktop with the RTX 2060, I have an entry with a field named “Video Engine Utilization” which shows me how much, in %, the video dec/enc is used right now. It is not there on the Jetson. Any way to get that information somehow? A command line maybe? That way I could clearly verify whether the hardware is used or not.

Hi,
There is individual hardware encoder/decoder on Jetson platfoms and the implementation is different from desktop GPUs. We have enabled hardware decoding. Please refer to development guide.

I am curious if that works for Jetson Agx Xavier nvidia dev kit also.
Based on the document I can see that I have to build FFMPEG for Jetson device but I could not build it on jetson successfully.
nvv4l2dec_init_decoder() works on general purpose ffmpeg?

That looks good. We’ll try those functions and see what happens.

Any way to see the stats in a console, especially the percent usage of the GPU and VEU (Video Engine Utilization)? The same we see in the nvidia-settings

Okay, I found it…

nvidia-settings -q VideoEncoderUtilization -q VideoDecoderUtilization

Only that’s not available on the Jetson AGX Xavier. Any idea why it is not available yet on the Jetsons?

Hi,
For system profiling on Jetson platforms, you may run sudo tegrastats.

RAM 2471/31919MB (lfb 7040x4MB) SWAP 0/15959MB (cached 0MB) CPU [0%@1190,1%@1190,2%@1190,0%@1190,off,off,off,off] EMC_FREQ 1%@1331 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 0% AO@41C GPU@41.5C Tdiode@45.5C PMIC@100C AUX@39.5C CPU@41.5C thermal@40.7C Tboard@42C GPU 0/9 CPU 311/328 SOC 1557/1556 CV 0/0 VDDRQ 467/466 SYS5V 1944/1951

I guess that will include the NVENC and NVDEC entries when in use. I’ll try that tomorrow.

we are only missing the driver from Nvidia … " libnvcuvid.so "
Let me know how to install ‘libnvcuvid.so’ on jetson agx xavier nvidia dev kit.

Hi,

The implementation is different on Jetson platforms. Please check
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide%2Fmultimedia.html%23wwpID0E0JB0HA

Thats only a bare description for function calls … we tried every possible way to get compile a code using the calls without success … eventually , I think its blind without a suitable reference may be with example.

Hi,
For multimedia usecases on Jetson platofrms, we support gstreamer and jetson_multimedia_api. Please check the documents:
https://docs.nvidia.com/jetson/l4t-multimedia/index.html
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide%2Faccelerated_gstreamer.html%23
There are hardware encoding/decoding blocks in Jetson chips so the implementation uses the blocks and leave GPU for other tasks.

Hi,
Fir video decoding, you can refer to 00_video_decode. After installation through SDKManager, the sample is in

/usr/src/jetson_multimedia_api/samples/00_video_decode

Online document is in

In the sample, decoded YUVs are in NvBuffer and we can call NvEGLImageFromFd() to get EGLImage:

/**
* Creates an instance of EGLImage from a DMABUF FD.
*
* @param[in] display    An \ref EGLDisplay object used during the creation
*                       of the EGLImage. If NULL, nvbuf_utils() uses
*                       its own instance of EGLDisplay.
* @param[in] dmabuf_fd  DMABUF FD of the buffer from which the EGLImage
*                       is to be created.
*
* @returns `EGLImageKHR` for success, `NULL` for failure
*/
EGLImageKHR NvEGLImageFromFd (EGLDisplay display, int dmabuf_fd);

And can perform CUDA operation:

/**
  * Performs CUDA Operations on egl image.
  *
  * @param image : EGL image
  */
static void
Handle_EGLImage(EGLImageKHR image)
{
    CUresult status;
    CUeglFrame eglFrame;
    CUgraphicsResource pResource = NULL;

    cudaFree(0);
    status = cuGraphicsEGLRegisterImage(&pResource, image,
                CU_GRAPHICS_MAP_RESOURCE_FLAGS_NONE);
    if (status != CUDA_SUCCESS)
    {
        printf("cuGraphicsEGLRegisterImage failed: %d, cuda process stop\n",
                        status);
        return;
    }

    status = cuGraphicsResourceGetMappedEglFrame(&eglFrame, pResource, 0, 0);
    if (status != CUDA_SUCCESS)
    {
        printf("cuGraphicsSubResourceGetMappedArray failed\n");
    }

    status = cuCtxSynchronize();
    if (status != CUDA_SUCCESS)
    {
        printf("cuCtxSynchronize failed\n");
    }

    if (eglFrame.frameType == CU_EGL_FRAME_TYPE_PITCH)
    {
        //Rect label in plan Y, you can replace this with any cuda algorithms.
        addLabels((CUdeviceptr) eglFrame.frame.pPitch[0], eglFrame.pitch);
    }

    status = cuCtxSynchronize();
    if (status != CUDA_SUCCESS)
    {
        printf("cuCtxSynchronize failed after memcpy\n");
    }

    status = cuGraphicsUnregisterResource(pResource);
    if (status != CUDA_SUCCESS)
    {
        printf("cuGraphicsEGLUnRegisterResource failed: %d\n", status);
    }
}

Please check the samples and see if it can be applied to your usecase. Thanks.

Hi,
And on r32.4.3, we have enabled hardware decoding in ffmpeg. Please check
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide%2Fmultimedia.html%23wwpID0E0JB0HA

Related posts:



The implementation is also based on jetson_multimedia_api.