I am just starting to use the Optical Flow NvOFFRUC sample app to do some testing of frame interpolation of high fps video. I am able to generate intermediate frames with no issues but when I build a video from the frames I get a slight flicker in the video. I have looked for more detailed documentation on the parameters for the interpolation but have not sorted it out yet. It seems the frame timestamps may have something to do with that but looking for help. I am using FFMPEG concat to rebuild the videos from the generated frames.
Hello @chris.marshall and welcome to the NVIDIA developer forums!
Great to see your interest in the Optical Flow SDK. I have informed some of the engineers involved about your question, hopefully they will find some time to help you along.
Thanks!
in looking a bit closer it seems that the interpolated frames are slightly lighter or darker than the frames used to create them, so when combined it looks like a flicker. Multiple passes generate interpolated frames that are then consistent with the other interpolated frames. So it seems like it is just the first pass. So if you then remove the original frames and just combine the generated frames it looks better, less flicker.
Hoping for some info on the above question.
Also, FRUC does not seem to use much of the GPU resources, is there a way to make it run faster? Will it use multiple GPUs in a system?
Hi @chris.marshall , sorry for the lag on answering.
First of all, note that FRUC leverages our Optical Flow Hardware Accelerator which is a dedicated unit to compute motion vectors on the GPU without taking any cycles from the CUDA cores or of the NVENC hardware. It is purposely built as a companion for resource intensive games or video applications. So low usage of other GPU resources when FRUC is used is actually expected and in fact desired behavior.
FRUC does not leverage multi GPU today.
Now to answer your flicker problem, we believe a possible reason could be that multiple calls of NvOFFRUCProcess are being made with same images.
To interpolate an intermediate frame, NvOFFRUCProcess API uses N and N+1 frames. Only N+1 frame is passed as parameter while NvOFFRUCProcess API uses an internally cached frame for frame N.
If a user calls NvOFFRUCProcess a second time with the same frame as input parameter, the new N frame in cache will be exactly the same as the new N+1 frame. In that case, instead of interpolating, you will create frames that are almost identical. Moreover, as interpolation is being computed with decimal numbers on 2 images, you might see some slight brightness/color changes in the newly calculated frame compared to the reference frame although you pass 2 identical frames to interpolate between. The video created by interleaving such frames with original frames will not show proper continuity of motions for objects across the frames and also have gradual brightness changes across frames that could be perceived as flickers.
Note that these effects could be even worse if calling NvOFFRUCProcess 3, 4 or more times.
The only way to currently workaround that problem is to run your interpolation process on the full video as many times as you want to create interpolated frames. Let’s say you’d like to achieve a 4x up conversion of your video with timestamps of 0 (cached reference N frame), 0.25, 0.5, 0.75 (for the 3x interpolated frames) and 1 (for the reference N+1 frame passed as parameter). Then you’ll have to go through the process of 0.25 for the entire video content, then run it with 0.5 for the entire video again and finally with 0.75 for the full video content. All these frames can then be combined to generate 4x up converted video. Same technique can be used for creating videos of 3x, 8x and so on.
We understand this is not an optimized workflow and will work on providing improvements to developer desiring to increase the framerate by more than 2x. In the meantime, at least you have a way to test the technology on your end.