Optical flow for Y16 images and improving quality

[user74794]

I’ve successfully used NVENC optical flow through VPI for quite some time. I now have an use case for a high-bit depth camera and optical flow. However, VPI only accepts NV12 frames in this case. Looking at some DRIVE OS docs and the libraries (libnvmedia.so) present on my L4T install, the underlying functions also seem to support 16-bit luma-only images.

Our use case is global motion estimation for 16-bit grayscale camera frames, preferrably without downquantization to 8 bits.

I have 2 questions:

1. Do you have any suggestions for our use case outlined above?
2. Could you point me towards possible methods for improving the quality of the flow vectors? Comparing the vector maps to ones from the newer architecture, there is a large difference. We already use forward-backward flow comparison to filter out bad vectors.

We currently use the Xavier NX, JetPack 4.6.2, VPI 1.1/1.2.

[AastaLLL]

Hi,

1. Currently, VPI dense optical flow only supports Y8 input.
You will need to downscale the image to 8 bits with NVENC’s dense optical flow.

2. The optical flow of Ampere GPU uses a new NVOFA hardware rather than NVENC.
So the difference comes from hardware implementation.

Thanks.

[user74794]

Thanks for the response.

Is there any other API available on Linux 4 Tegra which exposes the Y16 capability of NvMedia?

If there is, is there a benefit to using it in terms of quality, i.e. does the hardware use the full range or would it be downquantized internally anyway?

[AastaLLL]

Hi,

There are lots of VPI algorithms that can use 16-bit images as input.
For example bilateral filter:
https://docs.nvidia.com/vpi/algo_bilat_filter.html

But the optical flow can only support Y8.
Thanks.