disparity issue

Hi, everyone!

There are two things confused me very much, I wish that someone could me!

  1. When i want to get two disparity image, i find something interesting.
    1). Visionworks SGBM.
    If you use left image as a base image, and use right image to calculate a disparity image, you can get a right image;
    But if you use right image as a base image, and use left image to calculate a disparity image, you will get a very different image from left disparity image.

2). OpenCV SGBM
When i try opencv to do the same thing, the left disparity and the right disparity are very much similar.

So here is my problem, why SGBM in visionworks could get a different disparity image?

  1. Max value in disparity image problem.
    1). VisionWorks SGBM
    By setting min_disparity = 0 max_disparity = 64, then a disparity could be calculated, and the max value of this image is 255;
    2). OpenCV
    By doing the same thing, I have max value of disparity image is 64.

So why max value of disparity in visionwork is larger then 64?

Here are my code:
A part in stereo_matching.cpp

NVXIO_SAFE_CALL( vxQueryImage(left, VX_IMAGE_ATTRIBUTE_FORMAT, &format, sizeof(format)) );
        NVXIO_SAFE_CALL( vxQueryImage(left, VX_IMAGE_ATTRIBUTE_WIDTH, &width, sizeof(width)) );
        NVXIO_SAFE_CALL( vxQueryImage(left, VX_IMAGE_ATTRIBUTE_HEIGHT, &height, sizeof(height)) );

        main_graph_ = vxCreateGraph(context);
        NVXIO_CHECK_REFERENCE(main_graph_);

        //remap image to undistortion part
        vx_image left_remap = vxCreateVirtualImage(main_graph_, width, height, VX_DF_IMAGE_U8);
        NVXIO_CHECK_REFERENCE(left_remap);

        vx_image right_remap = vxCreateVirtualImage(main_graph_, width, height, VX_DF_IMAGE_U8);
        NVXIO_CHECK_REFERENCE(right_remap);

        left_remap_node_ = vxRemapNode(main_graph_, left, calib_params.remapTableL, VX_INTERPOLATION_BILINEAR, left_remap);
        NVXIO_CHECK_REFERENCE(left_remap_node_);

        right_remap_node_ = vxRemapNode(main_graph_, right, calib_params.remapTableR, VX_INTERPOLATION_BILINEAR, right_remap);
        NVXIO_CHECK_REFERENCE(right_remap_node_);

        // evaluate stereo
        vx_image disparity_short = vxCreateVirtualImage(main_graph_, width, height, VX_DF_IMAGE_S16);
        NVXIO_CHECK_REFERENCE(disparity_short);
 //
        // The SGM algorithm is now added as a node to the graph via the
        // nvxSemiGlobalMatchingNode().The input to the SGM node is previosuly
        // constructed left_gray and right_gray vx_images and the configuration
        // parameters. The output of the SGM node is the disparity_short image
        // that holds S16 fixed-point disparity values. The fixed-point values
        // have Q11.4 format (one sign bit, eleven integer bits and four
        // fractional bits). For the ease of further processing, we convert the
        // disparity map from fixed-point representation to U8 disparity
        // image. To do this, we drop the 4 fractional bits by right-shifting
        // the S16 values and then simply scale down the bit-width precision via
        // the vxConvertDepthNode().
        //

        semi_global_matching_node_ = nvxSemiGlobalMatchingNode(
            main_graph_,
            left_remap,
            right_remap,
            disparity_short,
            params.min_disparity,
            params.max_disparity,
            params.P1,
            params.P2,
            params.sad,
            params.ct_win_size,
            params.hc_win_size,
            params.bt_clip_value,
            params.max_diff,
            params.uniqueness_ratio,
            params.scanlines_mask,
            params.flags);
        NVXIO_CHECK_REFERENCE(semi_global_matching_node_);

        // convert disparity from fixed point to grayscale
        vx_int32 shift = 4;
        vx_scalar s_shift = vxCreateScalar(context, VX_TYPE_INT32, &shift);
        NVXIO_CHECK_REFERENCE(s_shift);
        convert_depth_node_ = vxConvertDepthNode(main_graph_, disparity_short, disparity, VX_CONVERT_POLICY_SATURATE, s_shift);
        vxReleaseScalar(&s_shift);
        NVXIO_CHECK_REFERENCE(convert_depth_node_);

        // verify the graph
        NVXIO_SAFE_CALL( vxVerifyGraph(main_graph_) );

        // clean up
#if 0
        vxReleaseImage(&left_gray);
        vxReleaseImage(&right_gray);
#else
        vxReleaseImage(&left_remap);
        vxReleaseImage(&right_remap);
#endif
        vxReleaseImage(&disparity_short);

A part form main.cpp

vx_image left = nvx_cv::createVXImageFromCVMat(g_context, left_image);
                NVXIO_CHECK_REFERENCE(left);
                vx_image right = nvx_cv::createVXImageFromCVMat(g_context, right_image);
                NVXIO_CHECK_REFERENCE(right);

                vx_image disparity = vxCreateImage(g_context, g_calib_params.calib_Sz[1], g_calib_params.calib_Sz[0], VX_DF_IMAGE_U8);
                NVXIO_CHECK_REFERENCE(disparity);

                std::unique_ptr<StereoMatching> stereo(
                    StereoMatching::createStereoMatching(
                        g_context, g_params, g_calib_params,
                        g_implementationType,
                        left, right, disparity));

                nvx::Timer procTimer;
                procTimer.tic();

                stereo->run();

                proc_ms = procTimer.toc();
                std::cout << "Display Time : " << proc_ms << " ms" << std::endl;

                nvx_cv::VXImageToCVMatMapper mapper(disparity, 0, NULL, VX_READ_AND_WRITE, VX_MEMORY_TYPE_HOST);
                output_disp_image = mapper.getMat().clone();

Is there anyone troubled by these issues,

Hi,

  1. Single direction stereo matching is common in the real application.
    It’s easy to know which source is from the left camera, which is on the right.
    Searching can be applied to only one direction to maximize the performance.

  2. Image value is between [0, 255]. Disparity value is [0, 64].
    But there is a scaling function in the stereo matching sample, which will increase disparities by k times for better display.
    You can turn it off to get the original value.

Hi AastaLLL:

Thanks for replying.

  1. Is there a non single direction stereo matching method which I can use, or how can I set the program to get a right disparity image by sending right_image as based image?

  2. OK, I will check the stereo matching sample again.
    So can I just normalize the output disparity image by multiplying ‘64/255’ to get origin disparity image?

BR!

Hi,

I checked the sample and find there is a “vxCreateScalar” function, and the result is multiplied by 4.

thanks

Hi AastaLL:

I have tried to set shift value as 1-5, I still can't get what a max value as 64.
vx_int32 shift = 4;
        vx_scalar s_shift = vxCreateScalar(context, VX_TYPE_INT32, &shift);
        NVXIO_CHECK_REFERENCE(s_shift);
        convert_depth_node_ = vxConvertDepthNode(main_graph_, disparity_short, disparity, VX_CONVERT_POLICY_SATURATE, s_shift);
        vxReleaseScalar(&s_shift);
        NVXIO_CHECK_REFERENCE(convert_depth_node_);

BR!

Hi,

Semi-Global Matching algorithm information is located at
VisionWorks API > Samples and Demos > Demo Applications > Stereo Matching Demo App of our document.
Please check it first.

  1. Change scanlines_mask can decide the searching direction:
    scanlines_mask
    Parameter: [integer value in range 0 to 255]
    Description: Bit-mask for enabling any combination of 8 possible directions. The lowest bit corresponds to “from-left-to-right” direction (NVX_SCANLINE_LEFT_RIGHT enumeration value). The second lowest bit corresponds to “from-top-left-to-bottom-right” direction (NVX_SCANLINE_TOP_LEFT_BOTTOM_RIGHT enumeration value), and so on. Default is 255.

  2. The disparity is smoothed. Please check P1 and P2 value:
    P1
    P2

    Parameter: [integer value greater than or equal to zero and less than or equal to 256]
    Description: Penalty parameters for SGBM algorithm. The larger the values, the smoother the disparity. P1 is the penalty on the disparity change by plus or minus 1 between neighbor pixels; P2 - by more than 1. The algorithm requires P2 > P1. Defaults are 8 and 109.

Hi,

  1. I had tried to change scanlines_mask, and used “from left to right” by set the value as 7, and set the value as 112 for “from right to left” direction, but the result dose not change too much. The output of left_disparity are quite good at both situation, but right_disparity are the same bad. Am I setting the wrong value?
min_disparity = 0
max_disparity = 64
P1 = 8
P2 = 108
sad = 5
bt_clip_value = 31
max_diff = 32000
uniqueness_ratio = 0
scanlines_mask = 112
flags = 2
ct_win_size = 0
hc_win_size = 1
min_disparity = 0
max_disparity = 64
P1 = 8
P2 = 108
sad = 5
bt_clip_value = 31
max_diff = 32000
uniqueness_ratio = 0
scanlines_mask = 7
flags = 2
ct_win_size = 0
hc_win_size = 1

Hi,

If possible, could you share your image for us checking?
Thanks.

Hi

I have the source image pair. but I don’t know how to share it with you;

Best regards

Hi

You can check “https://drive.google.com/open?id=0B4ACjvq2QXIyNlRHaFFjN3JqLTA” for right image and “https://drive.google.com/open?id=0B4ACjvq2QXIyUnZmZkNxOEwwdEU” for left image.

BR

Thanks.

Will update information to you later.

Hi

It could be grateful.

BR

Hi,

This source is a quite textureless. Not easy to have a good result.

For searching direction, you can check the value ‘scanlines_mask’.
Scanlines_mask value is between [0, 255] and each bit represents a direction.(total eight different searching direction)
For example, if you want to search for all possible orientations, set the value to 255.

For accuracy, set flag to 0 may help, which will turn-off cost filtration of SGBM algorithm.
For example, my setting is:

min_disparity = 0
max_disparity = 64
P1 = 8
P2 = 109
sad = 5
bt_clip_value = 31
max_diff = 32000
uniqueness_ratio = 0
scanlines_mask = 255
flags = 0
ct_win_size = 0
hc_win_size = 1

Hi.

By setting the same parameters as you did, the result dose not change much.

As you can see “https://drive.google.com/open?id=0B4ACjvq2QXIyQVpmSlhrZk83VnM” as left image, “https://drive.google.com/open?id=0B4ACjvq2QXIyeDRWeXFrSmZJM0U” as right image, “https://drive.google.com/open?id=0B4ACjvq2QXIyWXZwWjR5ZU40WEk” as left disparity, “https://drive.google.com/open?id=0B4ACjvq2QXIyck1IM3BBR0d1eTQ” as right disparity.

BR

Hi,

Please remember to switch the image order when calling stereo API.

For example,
Base = Right:

std::unique_ptr<StereoMatching> stereo(
  StereoMatching::createStereoMatching(
  context, params,
  implementationType,
  left, right, disparity));

Base = Left:

std::unique_ptr<StereoMatching> stereo(
  StereoMatching::createStereoMatching(
  context, params,
  implementationType,
  right, left, disparity));

Hi,

Of course I did the change you said.

The reason I want to get left_to_right disparity and right_to_left disparity is that because these two images are needed when I use a disparity filter provided by opencv , you can check “http://docs.opencv.org/trunk/db/d72/classcv_1_1ximgproc_1_1DisparityFilter.html#a3dc752f1760b375f1ea12173c1f07a9e

Hi,

We got correct disparity of the right image with the command mention in #16.
Could you recheck it?

Hi

Yes, I had tried it, and I checked it again, the result is quite different.

BR

Hi ,

Yes, the result is the same as before. Because I did the same thing before, if it’s just this little problem, I could be really happy, but it’s not…

BR