Illegal Memory Access Error when using VPI with DeepStream in Python

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
, ,
1

• Hardware Platform (Jetson Orin NX)
• DeepStream Version 6.3
• JetPack Version 5.1.2 [L4T 35.4.1]

• Issue Type(questions, bugs)
I’ve raised this issue on the DeepStream Forum and was directed here.

I am trying to utilize VPI for efficient image operations within a DeepStream pipeline using Python. However, there are very limited resources available on how to achieve this.

I attempted to replicate the C code in Python, referencing Paul Bridger’s post to work with the NvBufSurface C struct in Python. The overall process I implemented was: GStreamer buffer → CUDA EGL frame → CuPy array → VPI Image.

While I was able to obtain the VPI Image, I encountered an issue where calling cudart.cudaGraphicsUnregisterResource(graphics_resources) resulted in a cudaErrorIllegalAddress error after the pipeline ran for some time.

How can I resolve the illegal memory access error?

• How to reproduce the issue?
Please refer to this Github repository. app.py is a modified version of deepstream_imagedata-multistream_cupy.py of deepstream_python_appsv1.1.8.

Thank you for your assistance!

3

Hi,

It seems that you try to wrap the cupy array into a VPI image.

Could you help to verify if “GStreamer buffer → CUDA EGL frame → CuPy array” can work?
If the cudart.cudaGraphicsUnregisterResource(graphics_resources) can work correctly if not wrapped into VPI?

For CuPy ↔ VPI, please find the below topic for more info:

Thanks.

4

Thank you @AastaLLL

Yes, my goal is to convert a GStreamer buffer to a VPI Image. I am trying to wrap the CuPy array into a VPI Image because I could only find examples that convert a GStreamer buffer to a CuPy array without memory copying. Are there any other recommended ways of achieving this in Python?

“GStreamer buffer → CUDA EGL frame → CuPy array” works, and I am also able to wrap it into a VPI Image and perform rescaling with VPI.

I tried commenting out the VPI wrapping code:

        vpi_image = vpi.asimage(n_frame_gpu)
        with vpi.Backend.CUDA:
            output = vpi_image.rescale((vpi_image.width // 2, vpi_image.height // 2))
            output = output.convert(vpi.Format.BGR8)

→ did NOT observe illegal memory access error.

5

Hi,

Thanks for the info.

We are going to check this issue with the source in the GitHub you shared above.
Have you also tested this issue with the latest VPI 3.2 in the JetPack 6.2?

Thanks.

7

Not yet, due to considerations that some production devices have already been shipped, we would like to develop using the same JetPack version.

Thank you!

8

Hi,

We can reproduce the same error on JetPack 6.2 and are checking with our internal team.
Will share more info with you if we have further info.

Thanks.

9

Nice, look forward to hear from you soon.

Thank you!

10

Hi,

The app can terminate normally after applying the below change:
Please give it a try on your side as well.

diff --git a/app.py b/app.py
index 6bc960a..c524658 100644
--- a/app.py
+++ b/app.py
@@ -229,6 +229,7 @@ def tiler_sink_pad_buffer_probe(pad, info, u_data):
         print(f"{output.width}x{output.height} {output.format}")
 
     finally:
+        del vpi_image, output
         check_cudart_err(
             cudart.cudaGraphicsUnregisterResource(graphics_resources)
         )  # This should be called for proper cleanup, but causes illegal memory access error.

Thanks.

11

Hi,

I am still encountering the same error even after adding del vpi_image, output. Additionally, the results seem inconsistent. Sometimes, the pipeline completes successfully (even without adding del vpi_image, output) even when inputs are same.

Thank you!

12

Hi,

We test this on JetPack 6.2 without docker.
And the WAR does fix the issue.

Could you give it a try?

Thanks.

13

Sure, will give it a try.

Could you explain a bit what caused the illegal memory access error? And why adding del vpi_img will work? As Python has automatic garbage collection.

Thank you!

14

Hi,

The step is to ensure that VPI is not holding the wrapped resources when deleting them afterward.

Thanks.