What's the recommended way for integration with deepstream

Robotics & Edge Computing Holoscan General
1

I have a deepstream application runs on AGX Orin. Current design utilizes MIPI-CSI to connect the sensor directly, and nvarguscamerasrc for interface to the gstreamer.
Now I am considering using HSB for sensor connection, I am wondering what’s the best way to connect these to my gstreamer pipeline? I would like to utilize holoscan sdk as much as possible for the receiving, CSI/bayer and HW ISP.

2

Thanks for your interest in using the Holoscan ecosystem. Could you please let us know which camera you’re using?

3

Thanks for your reply.
Down the road, we will develop our camera sensor board, like we always do. One of the sensors we use is IMX565, for example.
But for starter, we will start with the Lattice evaluation kit + Leopard imaging IMX274 sensor. If we can get that working, transition to our own sensor should be easier.
Ideally, Nvidia could provide a gstreamer plugin similar to nvarguscamerasrc, that takes HSB camera stream as input, and connect to other gstreamer plugins seamlessly.

4

I’d be interested in this as well. I have an Arducam IMX477 camera connected to CSI2 (and possibly in the future GMSL2).

Does Holoscan complement DeepStream? Vice versa? Are they supposed to be used together?

Thanks.

-Tony

5

If I understood correctly, tt appears in L4T 38.2, the HSB support is integrated into lower level L4T components, so user space applications can stream the data from local CSI camera or CoE camera the same way,

6

Holoscan and DeepStream are different SDKs, but they can work together at the GStreamer boundary—DeepStream pipelines are GStreamer graphs, and Holoscan provides bridge operators that connect via standard appsrc / appsink.

In Holohub, the gst_to_holo app shows frames flowing GStreamer → Holoscan (GstSinkOp), and holo_to_gst shows Holoscan → GStreamer (GstSrcOp). You define the GStreamer side with a custom pipeline string; the bridge element must be named sink (into Holoscan) or src (out of Holoscan). That pattern lets you keep sensor/HSB capture and processing (CSI/Bayer, HW ISP, etc.) in Holoscan and hand off to DeepStream plugins (nvvideoconvert, nvstreammux, nvinfer, …) on the GStreamer side—or the other way around.

There isn’t a single nvarguscamerasrc-style plugin for HSB; the practical approach is compose Holoscan operators for the HSB path and connect to DeepStream through these bridges. Pay attention to caps and memory (e.g. RGBA vs NV12, host vs memory:CUDAMemory / NVMM) and use nvvideoconvert in the GStreamer string where DeepStream expects NVMM/NV12.

Relevant holohub paths: operators/gstreamer/, applications/gstreamer/gst_to_holo, applications/gstreamer/holo_to_gst.

Best

Gil