Extracting Motion Vectors from h.264 stream using NVDEC

Hi, to give context we’ve been experimenting the Jetson TX1 and we are using Gstreamer to access HW video decoder. We would like to access motion vectors that are exported video encoder. OMX is the Gstreamer plugin that use NV HW video decoder but currently is not exposing any way to access motion vectors. I started to read on OpenMax IL specification and motion vectors should be accessible.

QUESTION 1: Is Nvidia HW ( acceleration) video decoder implement the OpenMax IL interface to access motion vectors?

We would like to achieve this is done with this library:

but using Nvidia video decoder. In this library they used ffmpeg but my understanding is HW video decoder is only accessible using Gstreamer/OMX on Jetson TX1.

Is there a more native way to use NV video decoder NVDEC? ( Could you point me the API documentation)

Also, I was a bit surprised to read that HW video decoder is only accessible through Gstreamer on Jetson TX1.

On this site it’s indicated that ffmpeg can use nvidia HW video decoder on Maxwell GPU:

And I believe Jetson TX1 has a Maxwell GPU.

QUESTION 2: Could you clarify if FFMPEG is supported on Jetson?
Is just a matter of compiling FFMPEG with cuda supporte has described.

(Note my questions if form both TX1 and TX2.)

Thank you.

Hi DanMorin,
On Tegra series, we neither support extracting motion vector, nor ffmpeg with HW acceleration.

We have some posts requesting ffmpeg with HW acceleration but we are still evaluating it.
https://devtalk.nvidia.com/default/topic/1027734
https://devtalk.nvidia.com/default/topic/1002969
https://devtalk.nvidia.com/default/topic/1015708

For extracting motion vector, could you share what is the usecase to have it in normal video playback? It looks to be an advanced functionality for debugging?

Hi DaneLLL, thank you for your help. Regarding usecase, it’s for low cost ( work done by encoder) approximation of dense optical flow.

Thank you.

We are also very interested in this possible feature.

The goal is the same. Very low cost optical flow extraction on videos, for building intelligent cameras. The GPU will be free for doing other processing.

Thanks!!

+1 for us,

We are working with the TX2, does this platform supports motion vector extraction or is it also not possible?

Regards