How to integrate RGB-Depth camera in DeepStream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi all,

I wanted to know if it is possible to somehow integrate a depth camera in deepstream to do simple detection and tracking and maybe, measure the 3D distance between the recognized classes. I haven’t found much on the internet, do you know if there is a dedicated g-stream plugin? Otherwise do you have any ideas on how I can organize the pipeline so that it can read the data stream produced by a depth camera?

Thanks

3

Hi @tiberio.falsiroli,
Sorry for late! This is similar as DeepStream Using Depth Camera - #3 by mchi , we will support it in future.

Thanks!

4

Hi @tiberio.falsiroli,
Would you share a simple pipeline about how you will use Depth camera, so that we may could give some suggestion about how to use Depth camera with DS?

Thanks!

5

I would need the depth-camera to be able to calculate a three-dimensional distance of the objects recognized by the network. I would like to integrate it into a simple pipeline for detection (and tracking).

That is mi actual pipeline:

  1. Source bin created with function:
GstElement * source_bin = create_camera_source_bin(0, argv[1]);

where create_camera_source_bin is:

static GstElement *
create_camera_source_bin(guint index, gchar * uri) {
GstElement * bin = NULL;
GstCaps * caps = NULL;
gboolean ret = FALSE;
gchar bin_name[16] = {};
g_snprintf(bin_name, 15, “source-bin-%02d”, index);
bin = gst_bin_new(bin_name);
GstElement * src_elem = gst_element_factory_make(“v4l2src”, “src_elem”);
if (!src_elem) {
g_printerr(“Could not create ‘src_elem’\n”);
return NULL;
}
GstElement * cap_filter = gst_element_factory_make(“capsfilter”, “src_cap_filter”);
if (!cap_filter) {
g_printerr(“Could not create ‘src_cap_filter’\n”);
return NULL;
}
caps = gst_caps_new_simple(“video/x-raw”, “format”, G_TYPE_STRING, “NV12”,
“width”, G_TYPE_INT, 1280, “height”, G_TYPE_INT, 960,
“framerate”, GST_TYPE_FRACTION, 30, 1, NULL);
GstElement * nvvidconv1, * nvvidconv2;
GstCapsFeatures * feature = NULL;
nvvidconv1 = gst_element_factory_make(“videoconvert”, “nvvidconv1”);
if (!nvvidconv1) {
g_printerr(“Failed to create ‘nvvidcovn1’\n”);
return NULL;
}
feature = gst_caps_features_new(“memory:NVMM”, NULL);
gst_caps_set_features(caps, 0, feature);
g_object_set(G_OBJECT(cap_filter), “caps”, caps, NULL);
nvvidconv2 = gst_element_factory_make(“nvvideoconvert”, “nvvidconv2”);
if (!nvvidconv2) {
g_printerr(“Failed to create ‘nvvidcovn2’\n”);
return NULL;
}
g_object_set(G_OBJECT(nvvidconv2), “gpu-id”, 0, NULL);
gst_bin_add_many(GST_BIN(bin), src_elem, cap_filter, nvvidconv1, nvvidconv2, cap_filter, NULL);
//NVGSTDS_LINK_ELEMENT (src_elem, nvvidconv1);
//NVGSTDS_LINK_ELEMENT (nvvidconv1, nvvidconv2);
//NVGSTDS_LINK_ELEMENT (nvvidconv2, cap_filter);
if (!gst_element_link(src_elem, nvvidconv1)) {
g_printerr(“Failed to link ‘src_elem, nvvidcovn1’\n”);
return NULL;
}
if (!gst_element_link(nvvidconv1, nvvidconv2)) {
g_printerr(“Failed to link ‘nvvidcovn1, nvvidcovn2’\n”);
return NULL;
}
if (!gst_element_link(nvvidconv2, cap_filter)) {
g_printerr(“Failed to link ‘nvvidcovn2, cap_filter’\n”);
return NULL;
}
// NVGSTDS_BIN_ADD_GHOST_PAD (bin, cap_filter, “src”);
GstPad * gstpad = gst_element_get_static_pad(cap_filter, “src”);
gst_element_add_pad(bin, gst_ghost_pad_new(“src”, gstpad));
gst_object_unref(gstpad);
gchar device[64];
g_snprintf(device, sizeof(device), “/dev/video%d”, 0);
g_object_set(G_OBJECT(src_elem), “device”, device, NULL);
return bin;
}

After this, i link others plugin like nvinfer and tracker:

streammux, pgie, nvtracker, nvvidconv, nvosd, transform, sink, NULL

This pipeline allows me to read a stream of data from a usb camera. My goal would be to replace this usb camera (which allows me a two-dimensional view of the distance) with a depth-camera (in order to have a three-dimensional distance)