Disparity map with VPI in jetson Xavier AGX

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hi,
I’m starting to work with VPI and I am trying to test the StereoDisparityEstimator function.
My code is something as the following:

CHECK_STATUS(vpiStreamCreate(VPI_BACKEND_CUDA|VPI_BACKEND_PVA| VPI_BACKEND_VIC, &streamLeft));
CHECK_STATUS(vpiStreamCreate(VPI_BACKEND_CUDA|VPI_BACKEND_PVA| VPI_BACKEND_VIC, &streamRight));
CHECK_STATUS(vpiCreateStereoDisparityEstimator(VPI_BACKEND_CUDA, 1280, 720, VPI_IMAGE_FORMAT_U16, NULL, &payload));
     
     VPIStereoDisparityEstimatorParams params;
     params.windowSize   = 5;
     params.maxDisparity = 64;
     VPIConvertImageFormatParams cvtParams;
     vpiInitConvertImageFormatParams(&cvtParams);
     while(true) {
       capL >> matL;
       capR >> matR;
           
      remap(matL, matL, Left_Stereo_Map_x, Left_Stereo_Map_y, cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar());
       remap(matR, matR, Right_Stereo_Map_x, Right_Stereo_Map_y, cv::INTER_LINEAR, cv::BORDER_CONSTANT, cv::Scalar());
       cv::cvtColor(matL, matL, COLOR_BGR2GRAY);
       cv::cvtColor(matR, matR, COLOR_BGR2GRAY);
       matL.convertTo(matL, CV_16UC1);
       matR.convertTo(matR, CV_16UC1);
       leftVpi = mat2VPI(leftVpi, matL);
       rightVpi = mat2VPI(rightVpi, matR);
     
       CHECK_STATUS(vpiSubmitStereoDisparityEstimator(streamLeft, VPI_BACKEND_CUDA, payload, leftVpi, rightVpi, disparity, NULL, &params));
       CHECK_STATUS(vpiStreamSync(streamLeft));
     
     // vpi to opencv
       VPIImageData outData;
       CHECK_STATUS(vpiImageLock(disparity, VPI_LOCK_READ, &outData));
     
       cv::Mat outFrame(outData.planes[0].height, outData.planes[0].width, CV_16UC1, outData.planes[0].data, outData.planes[0].pitchBytes);
     
       double min, max;
       minMaxLoc(outFrame, &min, &max);
       outFrame.convertTo(outFrame, CV_8UC1, 255.0 / (max - min), -min);
       CHECK_STATUS(vpiImageUnlock(disparity));
    }

After that I show the image and my results are very bad. All the images are like the following image:

1
11280×720 400 KB

I don’t know what I’m doing wrong, any ideas?
Thanks in advance.

3

Hi,

Could you check if you feed the left and right image correctly?
And please adjust the searching range based on the view difference between images.

Thanks.

4

Hi, thanks for your answer.
I’m sure the left and right image are correct, I switched between them just in case, but I got the same results.
And I added the following code to check if the correction is correct:

      cv::Mat imgRectify = Mat::zeros(matL.rows, matL.cols*2+10, matL.type());
    
      matL.copyTo(imgRectify(Range::all(), Range(0, matL.cols)));
      matR.copyTo(imgRectify(Range::all(), Range(matR.cols+10, matR.cols*2+10)));
      
      //If it gets too large to fit on the screen, it is scaled down, by 2, to fit.
      if(imgRectify.cols > 1920){
          resize(imgRectify, imgRectify, Size(imgRectify.cols/2, imgRectify.rows/2));
      } 
      //To draw the lines in the rectified image
      for(int j = 0; j < imgRectify.rows; j += 16){
          Point p1 = Point(0,j);
          Point p2 = Point(imgRectify.cols*2,j);
          line(imgRectify, p1, p2, CV_RGB(255,0,0));
      }
      imshow("Rectified image", imgRectify);

With that I can see the image well remapped.

And sorry but I’m not sure I undestood your request, can you explain to me what you mean with “And please adjust the searching range based on the view difference between images.”?

Thanks.

5

Hi,

Sorry the for the nonclear statement.

There are some parameters used in the VPI Stereo Disparity Estimator.
Could you check if it need some update based on your stereo pair?
https://docs.nvidia.com/vpi/group__VPI__StereoDisparityEstimator.html#structVPIStereoDisparityEstimatorParams

Thanks.