VPI Stereo Disparity background (sky) noise

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
#1

Hi,

I’m currently running vpi 2.1 stereo disparity example from vpi_sample_02_stereo_disparity on Jetson (Jetpack 5.0.2).
The results have a lot of noise in the background – mainly in the sky.
The sky in this example image (input) is clear.

Input:

left
left1920×1200 118 KB

right
right1920×1200 124 KB

Output:
(CUDA backend, disp num 128, fixed blocksize, default confidence threshold and quality)

disp_vpi
disp_vpi1920×1080 400 KB

Expected:

disp_carnegie
disp_carnegie1920×1200 247 KB

The parameters I’ve tried in combination of:
[backend] CUDA, PVA-NVENC-VIC
[max disparity] 128, 256
[block size] 3, 5, 7
[confidence threshold] 32767, 49150
[quality] 4, 6, 8

All of the disparity results had the noise in the sky.

Tried adding smoothing filters like gaussian filter to try to minimize the noise, but has little affect.

Including the image produced from opencv stereo algorithm using the same sample set:

disp_opencv
disp_opencv1920×1200 206 KB

The left and right images are both rectified and captured with global shutter

Any idea what’s causing the issue here?

#3

Hi,

Do you apply the filter to the input stereo images or the disparity map?

Thanks.

#4

Hi,

No filter applied to input stereo images or the disparity map
Thank you

#5

Hi,

Please try to apply some smoothing filter to the input image to see if it helps.

Thanks.

#6

Hi, I have already tried in various sizes of filters, including aggressively smoothing but it could never get rid of the noise in the sky.

#7

Hi,

Below is what we got by feeding your image pair to the default 02-stereo_disparity sample.

confidence_cuda
confidence_cuda1380×862 203 KB

It looks like the confidence of the sky is much lower.
You can try to update the disparity value if the confidence value is not high enough in that region.

Thanks.

#8

Hi,

Thanks for giving it a try.
As mentioned above, I’ve tried different combinations of disparity values block sizes, confidence threshold, and quality to see if I can get a better result.
I was not able to get rid of the sky noise.

Thanks!

#9

Hi,

Have you tried this with other libraries like OpenCV?

If not, please give it a try.
This will tell us whether this is a limitation of the algorithm or something we can tune to get a better result.

Thanks.

#10

Hi,

Yes, I have! Please see the last image on the post.

It’s looking like a limitation of the algorithm. Can you please try tuning it?

#11

Hi @AastaLLL ,

Following up to see if you were able to make any progress on this.
Please let me know if you need me to try anything!

Thanks