VPI Stereo Disparity Real World Results

[user159893]

Is anyone getting good real-world results with the VPI Stereo Disparity?

I’m running Stereo Disparity using a Jetson Nano 4GB, and Python. The sample images produce a great depth map. But when I use images from my Waveshare Stereo Camera, the result is almost pure noise.

I am using a window of 5, and a max disparity of 64.

Any tips for getting a useable disparity would be greatly appreciated!

Thanks : )

[AastaLLL]

Hi,

Have you tried to adjust the parameter to see if it helps?

https://docs.nvidia.com/vpi/group__VPI__StereoDisparityEstimator.html#structVPIStereoDisparityEstimatorParams

For further suggestions, please share an image pair for your capture device with us first.
Thanks.

[user159893]

Hi AastaLLL, thanks for your reply!

I have tried manipulating the Window and Max Disparity parameters. I am using the Python API. I don’t see the quality and confidence parameters in the Python function.

I am manually adjusting the confidence threshold with Scale in confidence_map.convert. The noise level has decreased, but the result still seems pretty poor.

Here is my code:

scale = 1
window = 5
max_disparity = 256
confidence_threshold = 327670

left = cv2.imread('stereo_pair/left.jpg')
right = cv2.imread('stereo_pair/right.jpg')

with vpi.Backend.CUDA:
    left = vpi.asimage(left).convert(vpi.Format.Y16_ER, scale=scale)
    right = vpi.asimage(right).convert(vpi.Format.Y16_ER, scale=scale)

confidence_map = vpi.Image(left.size, vpi.Format.U16)
disparity = vpi.stereodisp(left, right, backend=vpi.Backend.CUDA, window=window, maxdisp=max_disparity) 

with vpi.Backend.CUDA:
    disparity = disparity.convert(vpi.Format.U8, scale=255.0 / (32 * max_disparity))
    left = left.convert(vpi.Format.U8)
    right = right.convert(vpi.Format.U8)

with disparity.rlock():
    disparity_color = cv2.applyColorMap(disparity.cpu(), cv2.COLORMAP_JET)
    disparity_color = cv2.cvtColor(disparity_color, cv2.COLOR_BGR2RGB)

with left.rlock():
    left = left.cpu()
    left = cv2.cvtColor(left, cv2.COLOR_BGR2RGB)

with right.rlock():
    right = right.cpu()
    right = cv2.cvtColor(right, cv2.COLOR_BGR2RGB)

with vpi.Backend.CUDA:
    confidence_map = confidence_map.convert(vpi.Format.U8, scale=255.0/confidence_threshold)

with confidence_map.rlock():
    confidence_map = confidence_map.cpu()

    mask = cv2.threshold(confidence_map, 1, 255, cv2.THRESH_BINARY)[1]
    mask = cv2.cvtColor(mask, cv2.COLOR_GRAY2BGR)

    disparity_color = cv2.bitwise_and(disparity_color, mask)

combined_image = np.hstack((left, right, disparity_color))
cv2.imwrite("stereo_pair/combined.jpg", combined_image)

As a new member, I can only upload one image at a time, so I combined the left, right, and disparity images.

[AastaLLL]

Hi,

Thanks for sharing the data.

We are able to separate the left, right, and the disparity image, also reproduce the issue.
Will share more information with you after further checking.

Thanks.

[user159893]

Thank you! Looking forward to hearing what you find.

[AastaLLL]

Hi,

The reason for the noisy result is that the VPI algorithm expects the rectified image pairs as input.
Please do the rectification first and then feed the rectified images into the stereo disparity estimator.

We have managed a simple rectification code with the source shared here.
But since there is no camera intrinsic info from our side, the rectified images tend to be distorted, which also affects the disparity map.

test.py (2.0 KB)

$ python3 test.py 
$ python3 /opt/nvidia/vpi1/samples/02-stereo_disparity/main.py cuda ./rectified_left.png ./rectified_right.png

You should get an idea result if rectifying the images with the accurate matrix.

Thanks.

[user159893]

AastaLLL, thank you so much! I completely overlooked rectifying my images first. This greatly improves my results.