Stereo disparity block matching using Vision Accelerator

[ynjiun]

• Hardware Platform (Jetson / GPU) Jetson AGX Xavier
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) 4.4

I read about AGX Xavier Vision Accelerator description mentioned that it can be used for stereo disparity block matching. Questions:

1. Do we have a demo in DeepStream SDK to show how it works?
2. does Vision Accelerator support cuda opencv, for example: cv::cuda::StereoBM? do we have example demo source code for enable Vision Accelerator when calling cv::cuda::createStereoBM?

Thank you very much for your help.

[mchi]

No DeepStream for now,

You can check doc - https://docs.nvidia.com/vpi/algo_stereo_disparity.html and vpi code under /opt/nvidia/vpi

[ynjiun]

@mchi

Thank you for the information.

Questions:

1. what’s the relationship between VPI and deepstream?
2. can we use VPI in the deepstream plugin development? for example, develop a deepstream transform-ip plugin calling VPI stereo interface using CUDA-EGL API to perform inplace computation without copying out the image?

Thank you.

[mchi]

DeepStrean is a SDK that provides many GST plugins to operate NVIDIA hw accelerators and also provides many typical reference sample for many typical smart-city pipeline/use case. VPI includes a set of APIs that provides to operate VIC/CUDA/PVA, it’s relative low level API.

can we use VPI in the deepstream plugin development?
Yes, you can check the VPI API introduction in the link

[ynjiun]

@mchi

Thank you for the info. I run the vpi_sample_02_stereo_disparity and I trace the code it sets the maxDisparity=64. But why the output disparity.png has pixel value (disparity) ranging from 0 to 255? should it be ranging from 0 to 64?

I check the code again it has a line to scale the min/max disparity to 255, then I print out the min/max for a testing left/right image: min=0 and max=2016, how come? shouldn’t the max be 64?

Please help to explain the output disparity how all the above associates to the maxDisparity=64. Thanks a lot.

[ynjiun]

another question regarding maxDisparity for vpi_sample_02_stereo_disparity: can we set a higher than 64? for example 128?

When I set it to 128, and recompile, then I got all flat disparity output (all in max for entire scene). If I set it back to 64 or lower (say 32, or 16), then the disparity looks normal again. Does this imply the maxDisparity is limited to 64? Please advise. Thanks.

[mchi]

Hi @ynjiun,
Sorry for late! What backend are you using?
Could you share me the repo?

Thanks!

[ynjiun]

No problem. I am using docker running on ubuntu 18.04 LTS with 1080Ti GPU for development. Once ready, I will port it over to AGX Xavier. When running I set cuda as the backend:

VPIBackend backendType = VPI_BACKEND_CUDA;

Thanks.

[mchi]

Could you share me the repo (modified code and steps)?

[ynjiun]

attached please find the modified code and left/right images for testing:
modified code: main.cpp (7.8 KB)
left image:

right image:

main.cpp line 154: params.maxDisparity = 64; //original code
make
./vpi_sample_02_stereo_disparity cuda flc.jpg frc.jpg
mv disparity_cuda.png disparity_cuda_64.png

The disparity image

looks reasonable.

Then modify the params.maxDisparity = 128;
main.cpp line 154: params.maxDisparity = 128; //modified code
make
./vpi_sample_02_stereo_disparity cuda flc.jpg frc.jpg
mv disparity_cuda.png disparity_cuda_128.png

Then the disparity image

doesn’t look right.

I hope these info could help you to duplicate what I encountered.

Thanks.

[amycao]

About can not change maxDisparity to value other than 64, reproed the issue, and we have internal bug 200680267 to track this issue.

I run the vpi_sample_02_stereo_disparity and I trace the code it sets the maxDisparity=64. But why the output disparity.png has pixel value (disparity) ranging from 0 to 255? should it be ranging from 0 to 64?

–> finally the disparity output convert from VPI_IMAGE_FORMAT_U16 to CV_8UC1,
cvOut.convertTo(cvOut, CV_8UC1, 255.0 / (max - min), -min); //from here
with each pixel 8 bits, that’s why the pixel value ranging from 0-255

I check the code again it has a line to scale the min/max disparity to 255, then I print out the min/max for a testing left/right image: min=0 and max=2016, how come? shouldn’t the max be 64?
Please help to explain the output disparity how all the above associates to the maxDisparity=64. Thanks a lot.

–> Will check internally, and get back to you.

[amycao]

I check the code again it has a line to scale the min/max disparity to 255, then I print out the min/max for a testing left/right image: min=0 and max=2016, how come? shouldn’t the max be 64?
Please help to explain the output disparity how all the above associates to the maxDisparity=64. Thanks a lot.

–> Will check internally, and get back to you.

the output is fixed-point, so you have to divide it by 2^5, the resulting float is bounded by 64.
In order to get the disparity to be in 0 to 63 (inclusive) range, divide output disparity by 32 and store it in a float.