NVIDIA Optical Flow - VapourSynth plugin (Motion compensation)

Greetings everyone, mainly NVOF developers.

I represent the community of digital video processing, using the open-source graph processing framework VapourSynth.
Some of the best algorithms we use rely heavily on motion estimation and compensation. We reconstruct current frame from its surrounding to gather information and achieve, for example, purely temporal denoising.

For years now we’ve been stuck with a slow and not very reliable block-matching motion estimation. So I and, I believe, the whole community are interested in having precise dense optical flow calculation tool.

I made a proof of concept NVOF plugin. It’s open-source and available here:
https://bitbucket.org/mystery_keeper/vapoursynth-nvof

So far it’s unoptimized, and there might bugs and flaws in my logic. Comments on the code and improvement suggestions are welcome.

I wanted to tell my full experience in this post, but as new user, I cannot post more than one link/image. I’ll try to tell more in the comments and outline my main concern here:

NVOF cannot be used for motion compensation without a confidence field. And the cost field it produces cannot be used as one.

I get impressive results in some frame areas and horrible errors in others. And masking by thresholding the cost field doesn’t really help to discern the pixels with good vectors from the ones with errors. The cost field actually looks weird!

I’m very interested in the improvement of NVOF. My plugin can already be used as a testing tool. And I’d be happy to help with testing. The fastest way to reach me is in Discord: Kittyfluff#6935

Thank you, and keep up your work.

Here is the VapourSynth script I use for testing.

nvof-test.vpy (3.3 KB)

Here’s a frame, that has both good vectors and errors. It shows, why the cost field doesn’t work as confidence field.

Here’s the flow in a perfectly still sequence!

To debug the visualization, I wrote a Python script that generates a video of a circle moving along an infinity sign trajectory. Here’s the script.

infinity-circle.py (1.0 KB)

And here’s a very weird flow I get for it.

No feedback from the NVOF team, huh?
Well, I haven’t been idle and wrote a (very slow) local correlation filter.
With this I can finally find and correct the motion compensation errors.
Really need to accelerate it though.
Maybe with OpenCL.

Another thing NVOF desperately needs is scene change detection.

I’m getting random BSODs when trying to use my filter.
Doesn’t seem to be a memory issue.
Can’t tell if its a problem in NVOF or in my code.

dude, have you tried to improve the optical flow by checking the consistency with Forward and Backward.

Finally got time and tried it.
It filtered out some good vectors.
It didn’t filter out some very bad vectors.
It significantly ramped up VRAM and RAM consumption and slowed down my filter.
It’s still unreliable, so I’ll just keep using correlation for now.

Also, noticed finally getting no vectors in perfectly still frames.
Likely there were improvements in drivers.
Still getting false vectors in flat backgrounds with a single moving object.