Various problems with VPI stereo disparity estimation algo

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
, ,
1

So, I have been trying to use VPI for stereo disparity estimation with CUDA backend. I have encountered two specific problems:

  1. The parameter maxDisparity of VPIStereoDisparityEstimatorParams struct cannot be set to a value greater than 64. The documentation for CUDA backend does not mention any limitation on the value of this parameter. When a higher value than 64 is used, the stream seems to execute, but the output image contains all zeros. How to solve this issue?
  2. I tried to extend the disparity estimation sample for a zed camera (I can share the script if needed) but encountered the error cudaErrorDeviceUninitialized. when vpiStreamSync(stream) was called. The explanation for this error type says that it usually occurs if there no context bound to the current thread. Using gdb, I found that zed-sdk launches its own threads when the camera is initialized. This somehow causes the context of the current thread to be lost. I created a context, following the KLT tracker example and called vpiContextSetCurrent(ctx) before every vpi operation to make sure that all operations occur in the same context, but still no luck. I have a vague idea that zed-sdk invokes cuda runtime library which maybe causing this problem to persist even after a context creation. I don’t know how to make the context persistent. Please help!
3

Hi,

1.

Could you share the windowSize you set with us?

2.

Suppose you should add vpiContextPush and vpiContextPop to save the context corresponding.
Could you check if this helps first?

Thanks.

4

Thanks for the quick response.

  1. windowSize is 5, unchanged from default value.
  2. Can you suggest a sample where context push and pull have been used? I want to be sure I am doing it right.
5

Hi @AastaLLL

I tried using vpiContextPush and vpiContextPop but still get the same error. Using vpiContextGetCurrent I obtained the active context just before the call to vpiStreamSync is made. I found that the active context is exactly what was created using vpiContextCreate at the beginning of main. Please see the log below:

Created context at: 0x5562798310
NVMEDIA_ARRAY: 53, Version 2.1
NVMEDIA_VPI : 156, Version 2.3
Current context while submitting to disparity estimator:0x5562798310
Current context while syncing stream:0x5562798310
[WARN ] 2020-11-10 09:45:17 src/private/mem/img/Mapping_NvMediaImage_CUDAPitch.cpp:151 (NvMediaEglStreamProducerGetImage:NVMEDIA_STATUS_TIMED_OUT)
[NvMediaEglStreamProducerGetImage:813] Producer get common failed: 3VPI_ERROR_INTERNAL: (cudaGraphicsResourceGetMappedEglFrame:cudaErrorDeviceUninitialized)
[WARN ] 2020-11-10 09:45:17 src/private/mem/img/Mapping_NvMediaImage_CUDAPitch.cpp:107 (cudaEGLStreamConsumerDisconnect:cudaErrorDeviceUninitialized)
Segmentation fault (core dumped)

So, it seems that cudaErrorDeviceUninitialized is not related to a missing context after all. Do you have any other ideas on how to proceed?

6

Hi,

Could you share us a sample so we can check it for you?
Thanks.

7

zed_vpi.cpp (9.4 KB)
CMakeLists.txt (2.5 KB)
@AastaLLL here it is. You will need a zed camera and zed sdk installed on your jetson for this to run. I hope that’s not a deal breaker.

8

@AastaLLL Update: The code works fine with pva backend when the input image is resized to (480,270). It also works with CPU although it is painfully slow. Only CUDA backend fails.

9

Hi,

Thanks for your example.

We try to reproduce this issue but the StereoDisparityEstimator with CUDA backend works correctly in our environment.
The input data is modified to use VPI assets instead.

Could you also give it a try? vpi.cpp (7.5 KB)

Thanks.

10

@AastaLLL
In the code, you have stripped away all the parts about the zed sdk (which uses cuda) and was causing the issues in the first place. Since you have removed the parts that were causing the trouble, the example works fine, as expected. The problem occurs only when another library that uses cuda, in this case zed sdk, is used along with VPI. Even then, if pva or cpu backend is used for sgm, everything works fine but when cuda backend is used for sgm, it doesn’t work. In summary, this does not address the problem I am facing and I hope I have explained the problem better now.

Any word on the maxDisparity parameter when using cuda backend? In the code you sent, if maxDisparity is increased to, for example, 128, you will find that the output shows a blank image with all pixels red when using cuda backend. As per the documentation, there is no limit on this parameter when using cuda, so this is either a bug or an undocumented limitation in the cuda backend. The cpu backend works just fine with any value of maxDisparity.

11

Hi,

We will share the maxDisparity information once we got a feedback from our internal team.

For the cudaErrorDeviceUninitialized error:
Sorry that we don’t have a ZED camera and this make it hard to reproduce in our environment.

Based on your implementation, it’s possible that the zed process(CPU) and VPI (GPU) access the same buffer at the same time.
Please noted that Jetson doesn’t support concurrent access from different processors.

So could you help to check if zed might access the buffer (for next frame?) when VPI is processing?
Since some of VPI functions is asynchronized and won’t wait for GPU return.

Thanks.

12

Hi jaiyamsharma,

Any update?

13

Hi,

Our internal team is working on the CUDA backend maxDisparity issue.
Will share more information with you once we have any progress.

Thanks.

1 Like
15

Hi @kayccc,
zed sdk is closed source, so I cannot exactly confirm what is going on behind the scenes when data is pulled from the camera. I found an opencv based version of their api which is open source and does not do anything weird behind the scenes. I will have a go with it over the weekend and report back.

16

@AastaLLL Thanks. I look forward to your response. Hopefully, this issue can be fixed by vpi 1.0 release scheduled in December.

17

Hi @kayccc I used opencv based capture from zed camera which does not use zed-sdk. CUDA backend works perfectly well with disparity estimation now. Thanks for helping me out. There is still the issue of maxDisparity which will hopefully be resolved by your internal team soon.

18

Thanks. The maxDisparity issue is checking internally.
Will update more information with you once we got any progress.

Thanks.

19

@AastaLLL @kayccc

So, to refresh your memory, I reported this issue in November and was told to wait while your team resolved it internally.

It is now February and I have been evaluating the newly released VPI 1.0 in JetPack 4.5. Unfortunately, the issue is still present as before, and has gotten even more severe. It is still not possible to set maxDisparity to greater than 64 in CUDA backend and the newly released docs for VPI1.0 still do not mention this as a limitation for CUDA backend (limitation is mentioned in PVA backend, so it is understandable). I say the problem is more severe now, because earlier the CPU backend used to work, as I had reported above but it doesn’t anymore .

So, three months of waiting and the issue is still not resolved. I hope you can resolve this soon.

1 Like
20

Hi,

Sorry for the late update.

Due to limited resources, this feature doesn’t be implemented in our VPI v1.0 release.
We are checking with our internal team for updating the document.
And will let you know once the ‘maxDisparity>64’ is supported.

Thanks.

1 Like
21

Thanks. As a suggestion, if NVIDIA shared the source of VPI, we could all contribute to development of these features rather than just filing issues. Hope to hear more about this soon.

22

Thanks for the suggestion.
But we don’t have a plan to share the VPI source currently.
Will let you once the feature is enabled.

Thanks.