VPI Stereo : zero disparity unless confidenceThreshold = 0 (confidenceType/NULL confidenceMap)

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

I am trying to run vpiSubmitStereoDisparityEstimator. I use OFA+PVA+VIC backends.

I have 2 questions.

1. How to obtain valid disparity images when using VPI_STEREO_CONFIDENCE_ABSOLUTE or RELATIVE

I observed that, except when confidenceThreshold is set to 0 (i.e., disabling disparities with confidence below the threshold), setting confidenceType to VPI_STEREO_CONFIDENCE_ABSOLUTE or RELATIVE results in a disparity image of all zeros. When I use the CUDA backend, a correct disparity image is produced in either case; I confirmed that the behavior differs by backend. The result was the same regardless of whether a confidenceMap was created.

Working code (produces correct disparity image):
vpiInitStereoDisparityEstimatorParams(&submitParams);
submitParams.confidenceType = VPI_STEREO_CONFIDENCE_ABSOLUTE; // or RELATIVE
submitParams.confidenceThreshold = 0;
vpiSubmitStereoDisparityEstimator(stream, backends, stereo, Left, Right, disparity, confidenceMap, &submitParams));

Non-working code (produces black / zero disparity image):
vpiInitStereoDisparityEstimatorParams(&submitParams);
submitParams.confidenceType = VPI_STEREO_CONFIDENCE_ABSOLUTE; // or RELATIVE
submitParams.confidenceThreshold = 32767; // Default value. Not set to 0.
vpiSubmitStereoDisparityEstimator(stream, backends, stereo, Left, Right, disparity, confidenceMap, &submitParams));

2. Effect of confidenceThreshold when confidence map flags are NULL

When I create a confidence map, the disparity map changes depending on confidenceThreshold (disparities below the threshold are invalidated). However, when the confidence map flags argument is NULL, I cannot obtain a valid disparity image unless confidenceThreshold = 0; otherwise I get a zero disparity image.

Working code (produces correct disparity image):
vpiInitStereoDisparityEstimatorParams(&submitParams);
submitParams.confidenceType = VPI_STEREO_CONFIDENCE_INFERENCE;
submitParams.confidenceThreshold = 32767; // Default value. Not set to 0.
vpiSubmitStereoDisparityEstimator(stream, backends, stereo, Left, Right, disparity, confidenceMap, &submitParams));

Non-working code (produces black / zero disparity image):
vpiInitStereoDisparityEstimatorParams(&submitParams);
submitParams.confidenceType = VPI_STEREO_CONFIDENCE_INFERENCE;
submitParams.confidenceThreshold = 32767; // Default value. Not set to 0.
vpiSubmitStereoDisparityEstimator(stream, backends, stereo, Left, Right, disparity, NULL, &submitParams));

Settings used in all tests

uint64_t backends = VPI_BACKEND_OFA | VPI_BACKEND_PVA | VPI_BACKEND_VIC;
VPIImageFormat InputFormat = VPI_IMAGE_FORMAT_Y8_ER_BL;
VPIImageFormat disparityFormat = VPI_IMAGE_FORMAT_S16;
vpiInitStereoDisparityEstimatorCreationParams(&createParams);
createParams.maxDisparity = 256;
vpiInitStereoDisparityEstimatorParams(&submitParams);
submitParams.numPasses = 1;
submitParams.windowSize = 1;

System details
NVIDIA Jetson AGX Orin
L4T 36.4.7 (JetPack 6.2.1)
Ubuntu 22.04.5 LTS
Kernel Version: 5.15.148-tegra
CUDA 12.6.68
OpenCV 4.8.0
OpenCV CUDA: NO
cuDNN: 9.3.0.75
VPI: 3.2.4
Vulkan: 1.3.204

Sorry for my poor English. Thank you.

1 Like
2

*** Please note that this reply is generated by LLM automatically ***

I’ll do my best to provide a comprehensive answer to your questions.

1. Obtaining valid disparity images when using VPISTEREOCONFIDENCEABSOLUTE or RELATIVE

You’ve observed that when using the OFA, PVA, or VIC backends, setting confidenceType to VPISTEREOCONFIDENCEABSOLUTE or RELATIVE results in a disparity image of all zeros, unless confidenceThreshold is set to 0. This behavior differs from the CUDA backend, which produces a correct disparity image in either case.

The reason for this behavior is likely due to the way the confidence threshold is applied in the OFA, PVA, and VIC backends. When confidenceType is set to VPISTEREOCONFIDENCEABSOLUTE or RELATIVE, the confidence threshold is used to filter out disparities with low confidence. However, if the confidence threshold is not set to 0, it’s possible that the threshold is too high, causing all disparities to be filtered out, resulting in a zero disparity image.

In contrast, the CUDA backend may have a different implementation of the confidence threshold, which allows it to produce a correct disparity image even when confidenceType is set to VPISTEREOCONFIDENCEABSOLUTE or RELATIVE.

To obtain valid disparity images when using VPISTEREOCONFIDENCEABSOLUTE or RELATIVE, you can try setting confidenceThreshold to a lower value, such as 0.1 or 0.01, to see if this allows more disparities to pass the confidence filter.

2. Effect of confidenceThreshold when confidence map flags are NULL

You’ve observed that when the confidence map flags argument is NULL, you cannot obtain a valid disparity image unless confidenceThreshold is set to 0. Otherwise, you get a zero disparity image.

This behavior is likely due to the fact that when the confidence map flags argument is NULL, the confidence threshold is not applied, and all disparities are considered valid. However, if confidenceThreshold is not set to 0, it’s possible that the threshold is too high, causing all disparities to be filtered out, resulting in a zero disparity image.

In this case, setting confidenceThreshold to 0 effectively disables the confidence filter, allowing all disparities to pass through.

Comparison of backends

The behavior you’ve observed suggests that there are differences in the implementation of the confidence threshold between the OFA, PVA, VIC, and CUDA backends. The CUDA backend appears to be more lenient in its application of the confidence threshold, allowing more disparities to pass through.

In contrast, the OFA, PVA, and VIC backends appear to be more strict in their application of the confidence threshold, requiring a lower threshold value to produce a valid disparity image.

System details

Your system details suggest that you are running on a NVIDIA Jetson AGX Orin with L4T 36.4.7, JetPack 6.2.1, and Ubuntu 22.04.5 LTS. Your VPI version is 3.2.4, and you are using OpenCV 4.8.0 with CUDA 11.6.8 and cuDNN 9.3.0.75.

Overall, the behavior you’ve observed suggests that there are differences in the implementation of the confidence threshold between the OFA, PVA, VIC, and CUDA backends. To obtain valid disparity images, you may need to adjust the confidenceThreshold value depending on the backend you are using.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

5

Thanks for the reply. However, it’s not an appropriate answer.

I also expected and understand that backends may apply parameters differently. My concern is whether the observed behavior is correct, and whether it can be fixed.

  1. You suggested lowering confidenceThreshold, but even setting it to 1 does not produce a correct disparity image. confidenceThreshold is an int32_t, so smaller fractional values cannot be supplied. Given that the CUDA backend works correctly with the same settings, isn’t this a bug?

  2. I also confirmed that it works correctly when using the CUDA backend. If you claim there are implementation differences, please explain concretely what those differences are. My goal is to use confidenceThreshold properly to suppress mismatches.

Thank you.

6

Hi,

  1. Based on the document, confidenceThreshold only support CUDA backend:

It’s recommended to add an error checker to the VPI function so it will show some error-related message:
Ex.

#define CHECK_STATUS(STMT)                                                                  \
    do                                                                                      \
    {                                                                                       \
        VPIStatus status = (STMT);                                                          \
        if (status != VPI_SUCCESS)                                                          \
        {                                                                                   \
            char buffer[VPI_MAX_STATUS_MESSAGE_LENGTH];                                     \
            vpiGetLastStatusMessage(buffer, sizeof(buffer));                                \
            std::ostringstream ss;                                                          \
            ss << "line " << __LINE__ << " " << vpiStatusGetName(status) << ": " << buffer; \
            throw std::runtime_error(ss.str());                                             \
        }                                                                                   \
    } while (0);

CHECK_STATUS(vpiInitStereoDisparityEstimatorCreationParams(&createParams));
  1. Do you have a complete source code that can reproduce the issue?

Thanks.

7

Hi, AastaLLL

  1. >Based on the document, confidenceThreshold only support CUDA backend. Where can I verify that information? According to the documentation, confidenceThreshold is supported by the CUDA backend and by the OFA+PVA+VIC backends.

    confidenceThreshold

  2. I am attaching the complete source. (The comments are written in Japanese. Sorry about that.)

    main.zip (3.6 KB)

Additionally, I have some newly discovered information to share.

  1. How to obtain valid disparity images when using VPI_STEREO_CONFIDENCE_ABSOLUTE or RELATIVE

I confirmed that with small image sizes like those used in the sample (470×280), ABSOLUTE or RELATIVE both work correctly. I am currently using 1632×1274 images; at that original size I get the error shown below, but it works correctly if I downsample the images.

line 234 VPI_ERROR_INTERNAL: Valid Right Padding range is [0, 255] inclusive, but input is 288
  1. Effect of confidenceThreshold when confidence map flags are NULL

It still hasn’t been resolved. If you know anything, please let me know.