NVIDIA Optical Flow SDK - How to interpret the output cost (confidence of the flow vectors)?

Professional Graphics and Rendering Video Technologies Video Codec and Optical Flow SDK
#1

I am interested in obtaining & interpreting the output cost (confidence) of the flow vectors that have been calculated via the Optical Flow SDK example application AppOfCuda. I have successfully modified the example to output the cost by:

  • Setting the enableOutputCost flag in initialisation parameters
  • Generating cost buffers via a call to nvOpticalFlow->CreateBuffers(NV_OF_BUFFER_USAGE_COST, NUM_COST_BUFFERS)
  • Executing NvOFBatchExecute whilst suppying the cost buffers which are then passed onto NvOF::Execute and m_NvOFAPI->GetAPI()->nvOFExecute(...)
  • Downloading the cost buffer data by calling costBuffers[i]->DownloadData(pCost.get()), which is then saved to an image

I am running the Optical Flow with NV_OF_MODE_OPTICALFLOW and the “slow” preset. I am interpreting the values as 32-bit unsigned integers. I’ve tried to split values to 16-bit unsigned ints and save out as separate images in case the x & y costs were interleaved but the high-order bytes seemed to always evaluate to 0.

When it comes to interpreting the result, I’m lost as the output cost looks a lot like one of the input images with some pixelated artefacts, except that the resulting image is at about a 1/16th of the full size (expected, as the motion vectors are 4x4).

The programming guide has the following piece of information:

Enable output cost: This flag enabled in case client desires to obtain the cost associated
with the generated flow vectors. <…> Higher cost value implies the flow vector to be less accurate and vice-versa.
But I found it insufficient.

Could you please give me an example of what an output cost map should look like or information on how the metric is supposed to be interpreted?

1 Like
#2

Hi,
The cost is the confidence level of the flow vector for each grid in the frame. The Cost implies how (in)accurate the flow vector is. It doesn’t have X and Y components and should be used as one whole value.
As explained in the documentation, cost is supposed to be used to decide whether to do further processing or take some further decision of the flow vector for the specific grid e.g. one may discard all flow vectors for the grids that have cost > X and use some other flow vectors instead. Hence, the down stream usage of the cost is very much dependent of the usage scenario and end solution being targeted. Therefore clients may need to do their own characterization.

Let me know if you have further queries.

Thanks.

#3

What you say, Mandar, makes sense. Yet I see the same thing as Justinas: About half of the pixels of the cost image are the downsampled input image (can’t remember if it’s the left or right). There are blocks of pixels that I can see could be interpreted as a measure of inaccuracy. That’s fine, but what about the rest?

If we are using the cost image to decide whether or not to accept a flow vector, how do we know the pixel we are looking at is a cost value or a downsampled input pixel?

To me, the “cost buffer” coming from AppOfCuda looks more like an internal scratch pad: originally it contained the downsampled input but was later used for some other temporary calculation.

#4

Hi,
Cost buffer does not contain any input image data. It is confidence value of motion vector. I am not sure about the reason behind interpreting the cost buffer as an image.
We need to try this and visualize the cost buffer and understand why cost buffer image is similar to input image.
Can you help us reproduce this issue internally.

Thanks.

1 Like
#5

Hello, I’m also trying to output the cost map, but I can’t output it successfully. The core dump always appears. Can you share your demo or tell me how to modify it?

#6

Hello, I am also trying to output the cost map, but I have been unable to output it successfully. The core dump always appears. Can you provide some samples?