VPI Stereo Disparity background (sky) noise

[stbnhan]

Hi,

I’m currently running vpi 2.1 stereo disparity example from vpi_sample_02_stereo_disparity on Jetson (Jetpack 5.0.2).
The results have a lot of noise in the background – mainly in the sky.
The sky in this example image (input) is clear.

Input:

Output:
(CUDA backend, disp num 128, fixed blocksize, default confidence threshold and quality)

Expected:

The parameters I’ve tried in combination of:
[backend] CUDA, PVA-NVENC-VIC
[max disparity] 128, 256
[block size] 3, 5, 7
[confidence threshold] 32767, 49150
[quality] 4, 6, 8

All of the disparity results had the noise in the sky.

Tried adding smoothing filters like gaussian filter to try to minimize the noise, but has little affect.

Including the image produced from opencv stereo algorithm using the same sample set:

The left and right images are both rectified and captured with global shutter

Any idea what’s causing the issue here?

[AastaLLL]

Hi,

Do you apply the filter to the input stereo images or the disparity map?

Thanks.

[stbnhan]

Hi,

No filter applied to input stereo images or the disparity map
Thank you

[AastaLLL]

Hi,

Please try to apply some smoothing filter to the input image to see if it helps.

Thanks.

[stbnhan]

Hi, I have already tried in various sizes of filters, including aggressively smoothing but it could never get rid of the noise in the sky.

[AastaLLL]

Hi,

Below is what we got by feeding your image pair to the default 02-stereo_disparity sample.

It looks like the confidence of the sky is much lower.
You can try to update the disparity value if the confidence value is not high enough in that region.

Thanks.

[stbnhan]

Hi,

Thanks for giving it a try.
As mentioned above, I’ve tried different combinations of disparity values block sizes, confidence threshold, and quality to see if I can get a better result.
I was not able to get rid of the sky noise.

Thanks!

[AastaLLL]

Hi,

Have you tried this with other libraries like OpenCV?

If not, please give it a try.
This will tell us whether this is a limitation of the algorithm or something we can tune to get a better result.

Thanks.

[stbnhan]

Hi,

Yes, I have! Please see the last image on the post.

It’s looking like a limitation of the algorithm. Can you please try tuning it?

[stbnhan]

Hi @AastaLLL ,

Following up to see if you were able to make any progress on this.
Please let me know if you need me to try anything!

Thanks

[AastaLLL]

Hi,

Sorry for the late update.

Could you share the script that generates OpenCV and VPI results?
So we can discuss this with the internal team to see if any parameter can be adjusted.

More, since there is a maximum limitation in the searching region.
Could you try the algorithm with a downscaled image pair?

Thanks.

[stbnhan]

Thanks for the reply!
The script I’m using is the sample application code from vpi2 pkg:
/opt/nvidia/vpi2/samples/02-stereo_disparity/

Except I added a few lines to configure the parameters mentioned above (confidence threshold, max disparity, quality, and window size)

// Set algorithm parameters to be used. Only values what differs from defaults will be overwritten.
VPIStereoDisparityEstimatorCreationParams stereoParams;
CHECK_STATUS(vpiInitStereoDisparityEstimatorCreationParams(&stereoParams));
stereoParams.maxDisparity = 256;
VPIStereoDisparityEstimatorParams stereoDispParams;
stereoDispParams.confidenceThreshold = 65280;
stereoDispParams.maxDisparity = stereoParams.maxDisparity;
stereoDispParams.quality = 6;
stereoDispParams.windowSize = 7;

I’ll attach the main.cpp anyway: main.cpp (16.5 KB)

I have tried down-sampling which seemed to have made a difference in CUDA backend, but not with PVA-NVENC-VIC backend. In the script, the latter backend option down-samples already so this would not be beneficial.

Left Image:

Right Image:

Disparity using CUDA backend (using downsampled input images):

Disparity using PVA-NVENC-VIC backend (using downsampled input images):

[stbnhan]

The images I shared above are compressed upon upload.
Including the original images here:
stereo_image.zip (1.8 MB)

[AastaLLL]

Hi,

Thanks for the sample.

Will get back to you later.

[stbnhan]

Hi @AastaLLL,

Following up on the progress on this topic.
Please let me know if you need anything else.

Thank you.

[AastaLLL]

Hi,

It seems that noise in the sky cannot be eliminated easily.

We need to check this issue with our internal team.
Could you share the script to generate the OpenCV output with the same stereo pair as well?

Thanks.

[stbnhan]

Hi @AastaLLL,

Thanks for looking into it!
Here’s the code block to generate opencv output:

void stereoDisparityOpenCV(cv::Mat &leftImage, cv::Mat &rightImage)
{
// Creating an object of StereoBM algorithm
cv::Ptrcv::StereoBM cv_stereo = cv::StereoBM::create();
cv_stereo->setNumDisparities(256);
cv_stereo->setBlockSize(25);
cv_stereo->setPreFilterType(1); // PREFILTER_XSOBEL;
cv_stereo->setPreFilterSize(7);
cv_stereo->setPreFilterCap(31);
cv_stereo->setTextureThreshold(10);
cv_stereo->setUniquenessRatio(15);
cv_stereo->setSpeckleRange(-1);
cv_stereo->setSpeckleWindowSize(0);
cv_stereo->setDisp12MaxDiff(-1);
cv_stereo->setMinDisparity(0);
cv::Mat cv_disparity;
cv_stereo->compute(leftImage, rightImage, cv_disparity);
cv::normalize(cv_disparity, cv_disparity, 0, 255, cv::NORM_MINMAX, CV_8U);
cv::imwrite(“disparity_cv.jpg”, cv_disparity);
}

Thank you.

[stbnhan]

Hi @AastaLLL,

Following up if you have any updates from your team.
As always, please let me know if you need any more info.

Thank you.

[AastaLLL]

Hi,

This issue can be improved with our new parameter in the next release.

For example, we can get a much better result if setting the uniqueness value=0.8.
(Best output still needs some parameter tuning)

Uniqueness=0.8

Thanks.

[stbnhan]

Hi @AastaLLL,

This looks very promising! Thank you for fixing this.
What does “uniqueness” do in the algorithm?
Do you have a timeline for the next release yet?

Thank you.