Accessing an image in shared CPU/GPU memory between C++ and Python


Hi, I am using the zed2i stereo camera which has multiple data sources: images (standard full HD @ 30 FPS), a depth map (essentially another image at the same resolution/rate), and sensor data (the camera comes with an interesting set of sensors).

We’ve built a very nice computer vision pipeline that’s running on our Jetson hardware (both AGX and NX modules), doing cool stuff like people detection, tracking, pose, etc., and being fed by this cameras images.

The problem: now I want to process the camera’s depth map in an accelerated fashion as well, and don’t know how to do it, at least not fast …

Image Capture in Python from the Zed2i

Currently, the capture stage of the pipeline uses dusty’s jetson.utils (awesome library, THANK YOU) VideoInput in Python, which seems to pull the images from the camera into GPU memory just fine. …SIDE NOTE: it seems to use a gstreamer approach under the hood that makes use of an argussrc to do the actual capture … is this still state of the art? …

In any case, the capture code looks like this in Python:

        self.stereoFrame = self.jetsonutilsSource.Capture(format="rgb8")
                    (0, 0, self.width, self.height),

This works great (is this best WTDI?), and my pipeline can run quickly against the images thus captured.

Depth Image Capture?

Now, I’d like to do the same for the depth image. For that it seems like I’ll need to use the Zed SDK API , which is probably for the best anyway since it’s purpose-built to work with the camera. It has both C++ and Python bindings, but Python (sadly) supports only capturing to CPU mem and not GPU (C++ can do both).

So what I’d like to do is write a capture module that captures images and depth (and even sensor data for that matter) into GPU memory, and then have my Python pipeline access said buffers (note: I can’t rewrite the pipeline in C++, don’t have the time).

Questions: how do I do this? Or is there a better way I am missing?

Things we’re thinking about:

  • Forking Jetson Utils and hacking it to use a custom gstreamer src/string that can do this. Note: stereolabs supports a custom gstreamer source, but if you look at how it captures under the hood it captures to CPU memory and then does a copy into GST memory… so could I hack the ‘argus’ source to do this?

  • Using the Zed Python SDK to capture into CPU memory and do a copy (which will be slow… in the Python SDK comments they even say “for performance, use C++”) … doable I suppose, but seems like a last resort…

  • Using Cython to simply call a small Python/C++ module using an approach that links with the Zed C++ SDK on the C++ side) that does the capture in C++ and returns the pointer to shared GPU/CPU memory. Since the Cython shared library .so is loaded into the python address space and run from Python, it seems like the pointer would be safe and usable…

Anyway, it just seems like it should be not-impossible to integrate a camera SDK that is built to work with Jetson out of the box and capture images to the GPU with a Python pipeline that also understands all things GPU…

Thanks if you’ve read this far … appreciate any help!

We support UVC driver by default, please check Jetson Nano FAQ

Q: I have a USB camera. How can I launch it on Jetson Nano?

jetson-utils is also based on UVC driver and v4l2. The format of depth camera can be special and may not be included. If you can run v4l2-ctl command to capture frames from the depth camera, you should be able to add the format to jetson-utils and give it a try.

DaneLLL, thx for the reply!

Camera is accessible and images can be grabbed via UVC driver, but depth map and sensors are not accessible it seems (see below).

I understand that Nvidia can’t support all the native camera drivers/SDKs out there, so it completely makes sense that your focus is v4l2/UVC.

As it looks like using the native SDK will be the only way forward (which is what I suspected), my hope was that the community here might be of help in assessing the possible approaches to capturing in C++ and using in Python that I mentioned based on possible experience they’ve had.

Hopefully it’s not off-topic for this forum, as it’s possible that others may be facing a similar challenge using zed or other cams with similar capabilities.


Driver Info:
Driver name : uvcvideo
Card type : ZED 2i: ZED 2i
Bus info : usb-0000:00:14.0-1
Driver version : 5.11.12
Capabilities : 0x84a00001
Video Capture
Metadata Capture
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Extended Pix Format
Media Driver Info:
Driver name : uvcvideo
Model : ZED 2i: ZED 2i
Serial : OV0001
Bus info : usb-0000:00:14.0-1
Media version : 5.11.12
Hardware revision: 0x00000100 (256)
Driver version : 5.11.12
Interface Info:
ID : 0x03000002
Type : V4L Video
Entity Info:
ID : 0x00000001 (1)
Name : ZED 2i: ZED 2i
Function : V4L2 I/O
Flags : default
Pad 0x01000007 : 0: Sink
Link 0x02000010: from remote pad 0x100000a of entity ‘ZED 2i’: Data, Enabled, Immutable
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
Width/Height : 2560/720
Pixel Format : ‘YUYV’ (YUYV 4:2:2)
Field : None
Bytes per Line : 5120
Size Image : 3686400
Colorspace : sRGB
Transfer Function : Rec. 709
YCbCr/HSV Encoding: ITU-R 601
Quantization : Default (maps to Limited Range)
Flags :
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 2560, Height 720
Default : Left 0, Top 0, Width 2560, Height 720
Pixel Aspect: 1/1
Selection Video Capture: crop_default, Left 0, Top 0, Width 2560, Height 720, Flags:
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 2560, Height 720, Flags:
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0
brightness 0x00980900 (int) : min=0 max=8 step=1 default=4 value=4
contrast 0x00980901 (int) : min=0 max=8 step=1 default=4 value=4
saturation 0x00980902 (int) : min=0 max=8 step=1 default=4 value=4
hue 0x00980903 (int) : min=0 max=11 step=1 default=0 value=0 flags=inactive
white_balance_temperature_auto 0x0098090c (bool) : default=1 value=1
white_balance_red_component 0x0098090e (int) : min=1024 max=4095 step=16 default=1024 value=1024 flags=inactive
white_balance_blue_component 0x0098090f (int) : min=1024 max=4095 step=16 default=1024 value=1024 flags=inactive
gamma 0x00980910 (int) : min=1 max=500 step=1 default=100 value=100
gain 0x00980913 (int) : min=0 max=8 step=1 default=4 value=4
power_line_frequency 0x00980918 (menu) : min=0 max=2 default=1 value=1
0: Disabled
1: 50 Hz
2: 60 Hz
hue_auto 0x00980919 (bool) : default=1 value=1
white_balance_temperature 0x0098091a (int) : min=2800 max=6500 step=100 default=4600 value=4600 flags=inactive
sharpness 0x0098091b (int) : min=0 max=8 step=1 default=0 value=0

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.