Inconsistencies in Optical Flow Results with NVIDIA DeepStream

[pedroe]

Hello everyone,

I am encountering some unusual behavior and I need your help to understand what is happening. I am working with NVIDIA DeepStream and have been using the nvof plugin to calculate optical flow on a series of videos.

The command I am using is as follows:

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_qHD.mp4 ! decodebin ! nvvideoconvert gpu-id=0 ! video/x-raw,format=NV12 ! nvvideoconvert ! nvstreammux0.sink_0 nvstreammux gpu-id=0 name=nvstreammux0 batch-size=1 batched-push-timeout=1000000 width=400 height=300 ! nvof dump-of-meta=1 preset-level=2 gpu-id=0 ! nvvideoconvert ! fpsdisplaysink sync=false

This command should generate files with the optical flow in the format nvopticalflow0_400x300_BS-0-FR-XXXX.mv. However, I am noticing that the optical flows generated are relatively similar, but clearly different, when executed on different GPUs.

I have attached an image depicting the flows for a specific frame (the 10th in the sequence) to illustrate the issue:

Furthermore, on some occasions, I have been able to reproduce this issue even on the same machine, simply by running the command twice. Although in this case, the issue is harder to reproduce than when running it on two different machines, certainly.

This leads me to believe that either a) the optical flow algorithm is not completely deterministic, or b) there is something in the pipeline, perhaps in the nvstreammux element, that makes the algorithm non-reproducible.

Does anyone have any idea what could be happening or has experienced something similar? Any help or direction would be greatly appreciated.

P.S. At least the number of output optical flow files () is the same in every execution, the contents of the respective corresponding files between executions are the ones that certainly differ.

Thank you in advance.

Deepstream version 6.1
GPUs RTX 3060 & RTX 2070

[Fiona.Chen]

I can reproduce the differences between different GPUs. We will investigate the issue.

[Fiona.Chen]

@pedroe Can you tell us which tool you are using to view the dumped mv files?

[pedroe]

Blockquote @pedroe Can you tell us which tool you are using to view the dumped mv files?

Sure! In fact I just made a small python script, flow2matplotlib.py, for that. Here it is:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import numpy as np
import matplotlib.pyplot as plt
import sys

def flow_cam_as_numpy(fname, H, W):
    # Read and return file data as numpy array (int16 type):
    with open(fname, "rb") as f:
        data = np.fromfile(f, dtype=np.int16)    
    assert data.size == H * W * 2
    reshaped_data = data.reshape(H, W, 2)
    return reshaped_data

if __name__ == '__main__':
    fname, W, H = sys.argv[1], int(sys.argv[2])//4, int(sys.argv[3])//4
    npflow = flow_cam_as_numpy(fname, H, W)
    x, y = np.meshgrid(np.arange(0, W), np.arange(0, H))
    fig, ax = plt.subplots()
    quiver = ax.quiver(x, y, npflow[:, :, 0], -npflow[:, :, 1], 
                       scale=100, scale_units='xy')
    ax.invert_yaxis()
    plt.show()

Then you just call it this way:

./flow2matplotlib.py nvopticalflow0_400x300_BS-0-FR-0010.mv 400 300

(The script does not convert int16 to float, it just uses a sensible scale in the ax.quiver call, in order to obtain a reasonably interpretable graphical output).

BTW, I noticed that in my previous post, I sent you my visualization for frame 10 of sequence /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4, instead of /opt/nvidia/deepstream/deepstream/samples/streams/sample_qHD.mp4.

So the correct commands to reproduce and visualize the problems would be:

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! decodebin ! nvvideoconvert gpu-id=0 ! video/x-raw,format=NV12 ! nvvideoconvert ! nvstreammux0.sink_0 nvstreammux gpu-id=0 name=nvstreammux0 batch-size=1 batched-push-timeout=1000000 width=400 height=300 ! nvof dump-of-meta=1 preset-level=2 gpu-id=0 ! nvvideoconvert ! fpsdisplaysink sync=false

and then:

./flow2matplotlib.py nvopticalflow0_400x300_BS-0-FR-0010.mv 400 300

Which leads to this visualization:

[Fiona.Chen]

@pedroe

Sure! In fact I just made a small python script, flow2matplotlib.py, for that. Here it is:

Thank you for sharing the script. We are investigating this issue. Will be back when there is any progress.

[Fiona.Chen]

@pedroe

We have confirmed that different generation of GPUs generate different flow. Each generation of OFA will have some improvements over last generation. So, It is expected behavior.