USB CAM problem

Hi,
I am trying to use USB camera on Xavier NX. I can get and show frame after modifying deepstream_test3_app.c with the method mentioned in this forum How to run deepstream-test3 with usb camera.
But I also have the choppy frame output problem, and someone mention that “should create two cap_filter in the implementation. One is for v4l2src and the other is for nvvideoconvert.”
However, I have no idea how to implement it. Is there any related tutorial or example?
Suggestions are highly appreciated.

Hi,
Please refer to this sample:
Use YUY2 format as input - #11 by DaneLLL

If your source supports UYVY, you may use nvv4l2camerasrc to get better performance.

DS5.0.1 is released. If you are using DS5.0, we would suggest do upgrade.

Hi, my camera input pixel format is Z16, and follow the method I mentioned, I can show the frame output.
So, for the choppy frame problem, I just need to follow the sample you provide and modify the program?
Thanks.

After reading Use YUY2 format as input - #11 by DaneLLL,
I also have a question that why should we need to create three caps, can I know the purpose of each cap?

Hi,
Z16 is a special format and is not listed in gst-inspect-1.0 v4l2src. Please check if you can run this:

$ export DISPLAY=:0
$ gst-launch-1.0 v4l2src ! video/x-raw,width=1920,height=1080,framerate=60/1 ! videoconvert ! video/x-raw,format=I420 ! xvimagesink

You need to change width, height, framerate to fit your source.

Following you command, I run

gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw,width=640,height=480,framerate=30/1 ! videoconvert ! video/x-raw,format=I420 ! xvimagesink

and got this error,

Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

And here is the device inform

Index       : 0
Type        : Video Capture
Pixel Format: 'Z16 '
Name        : 16-bit Depth
	Size: Discrete 480x640
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 768x1024
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 480x1024
		Interval: Discrete 0.033s (30.000 fps)

Index       : 1
Type        : Video Capture
Pixel Format: ''
Name        : 20204746-0000-0010-8000-00aa003
	Size: Discrete 800x600
		Interval: Discrete 0.033s (30.000 fps)
	Size: Discrete 1280x720
		Interval: Discrete 0.033s (30.000 fps)

Hi, @DaneLLL,
After following your suggestion to modify deepstream-test1, I only removed caps_uyvy part because the input format is different.
Here is my program.

#include <gst/gst.h>
#include <glib.h>
#include <stdio.h>
#include "gstnvdsmeta.h"

#define MAX_DISPLAY_LEN 64

#define PGIE_CLASS_ID_VEHICLE 0
#define PGIE_CLASS_ID_PERSON 2

/* The muxer output resolution must be set if the input streams will be of
 * different resolution. The muxer will scale all the input frames to this
 * resolution. */
#define MUXER_OUTPUT_WIDTH 640
#define MUXER_OUTPUT_HEIGHT 480

/* Muxer batch formation timeout, for e.g. 40 millisec. Should ideally be set
 * based on the fastest source's framerate. */
#define MUXER_BATCH_TIMEOUT_USEC 4000000

#define NVVIDCONV

gint frame_number = 0;
gchar pgie_classes_str[4][32] = { "Vehicle", "TwoWheeler", "Person",
  "Roadsign"
};

/* osd_sink_pad_buffer_probe  will extract metadata received on OSD sink pad
 * and update params for drawing rectangle, object information etc. */

static GstPadProbeReturn
osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
    gpointer u_data)
{
    GstBuffer *buf = (GstBuffer *) info->data;
    guint num_rects = 0; 
    NvDsObjectMeta *obj_meta = NULL;
    guint vehicle_count = 0;
    guint person_count = 0;
    NvDsMetaList * l_frame = NULL;
    NvDsMetaList * l_obj = NULL;
    NvDsDisplayMeta *display_meta = NULL;

    NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);

    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
        NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);
        int offset = 0;
        for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
                l_obj = l_obj->next) {
            obj_meta = (NvDsObjectMeta *) (l_obj->data);
            if (obj_meta->class_id == PGIE_CLASS_ID_VEHICLE) {
                vehicle_count++;
                num_rects++;
            }
            if (obj_meta->class_id == PGIE_CLASS_ID_PERSON) {
                person_count++;
                num_rects++;
            }
        }
        display_meta = nvds_acquire_display_meta_from_pool(batch_meta);
        NvOSD_TextParams *txt_params  = &display_meta->text_params[0];
        display_meta->num_labels = 1;
        txt_params->display_text = g_malloc0 (MAX_DISPLAY_LEN);
        offset = snprintf(txt_params->display_text, MAX_DISPLAY_LEN, "Person = %d ", person_count);
        offset = snprintf(txt_params->display_text + offset , MAX_DISPLAY_LEN, "Vehicle = %d ", vehicle_count);

        /* Now set the offsets where the string should appear */
        txt_params->x_offset = 10;
        txt_params->y_offset = 12;

        /* Font , font-color and font-size */
        txt_params->font_params.font_name = "Serif";
        txt_params->font_params.font_size = 10;
        txt_params->font_params.font_color.red = 1.0;
        txt_params->font_params.font_color.green = 1.0;
        txt_params->font_params.font_color.blue = 1.0;
        txt_params->font_params.font_color.alpha = 1.0;

        /* Text background color */
        txt_params->set_bg_clr = 1;
        txt_params->text_bg_clr.red = 0.0;
        txt_params->text_bg_clr.green = 0.0;
        txt_params->text_bg_clr.blue = 0.0;
        txt_params->text_bg_clr.alpha = 1.0;

        nvds_add_display_meta_to_frame(frame_meta, display_meta);
    }

    g_print ("Frame Number = %d Number of objects = %d "
            "Vehicle Count = %d Person Count = %d\n",
            frame_number, num_rects, vehicle_count, person_count);
    frame_number++;
    return GST_PAD_PROBE_OK;
}

static gboolean
bus_call (GstBus * bus, GstMessage * msg, gpointer data)
{
  GMainLoop *loop = (GMainLoop *) data;
  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_EOS:
      g_print ("End of stream\n");
      g_main_loop_quit (loop);
      break;
    case GST_MESSAGE_ERROR:{
      gchar *debug;
      GError *error;
      gst_message_parse_error (msg, &error, &debug);
      g_printerr ("ERROR from element %s: %s\n",
          GST_OBJECT_NAME (msg->src), error->message);
      if (debug)
        g_printerr ("Error details: %s\n", debug);
      g_free (debug);
      g_error_free (error);
      g_main_loop_quit (loop);
      break;
    }
    default:
      break;
  }
  return TRUE;
}

int
main (int argc, char *argv[])
{
  GMainLoop *loop = NULL;
  GstElement *pipeline = NULL, *source = NULL,
      *streammux = NULL, *sink = NULL, *pgie = NULL, *nvvidconv = NULL,
      *nvosd = NULL;
  GstElement *conv1 = NULL, *conv2 = NULL, *nvconv = NULL;
  GstCaps *caps_uyvy = NULL, *caps_nv12_nvmm = NULL, *caps_nv12 = NULL;
  GstCapsFeatures *feature = NULL;
#ifdef PLATFORM_TEGRA
  GstElement *transform = NULL;
#endif
  GstBus *bus = NULL;
  guint bus_watch_id;
  GstPad *osd_sink_pad = NULL;

  /* Standard GStreamer initialization */
  gst_init (&argc, &argv);
  loop = g_main_loop_new (NULL, FALSE);

  /* Create gstreamer elements */
  /* Create Pipeline element that will form a connection of other elements */
  pipeline = gst_pipeline_new ("dstest1-pipeline");

  /* Source element for reading from the file */
  source = gst_element_factory_make ("v4l2src", "v4l2-source");
#ifdef NVVIDCONV
  conv1 = gst_element_factory_make ("nvvidconv", "conv1");
  conv2 = gst_element_factory_make ("nvvidconv", "conv2");
#else
  conv1 = gst_element_factory_make ("videoconvert", "conv1");
#endif
  nvconv = gst_element_factory_make ("nvvideoconvert", "nvconv");

  /* Create nvstreammux instance to form batches from one or more sources. */
  streammux = gst_element_factory_make ("nvstreammux", "stream-muxer");

  if (!pipeline || !streammux) {
    g_printerr ("One element could not be created. Exiting.\n");
    return -1;
  }

  /* Use nvinfer to run inferencing on decoder's output,
   * behaviour of inferencing is set through config file */
  pgie = gst_element_factory_make ("nvinfer", "primary-nvinference-engine");

  /* Use convertor to convert from NV12 to RGBA as required by nvosd */
  nvvidconv = gst_element_factory_make ("nvvideoconvert", "nvvideo-converter");

  /* Create OSD to draw on the converted RGBA buffer */
  nvosd = gst_element_factory_make ("nvdsosd", "nv-onscreendisplay");

  /* Finally render the osd output */
#ifdef PLATFORM_TEGRA
  transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
#endif
  sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");

  if (!source || !pgie
      || !nvvidconv || !nvosd || !sink) {
    g_printerr ("One element could not be created. Exiting.\n");
    return -1;
  }

#ifdef PLATFORM_TEGRA
  if(!transform) {
    g_printerr ("One tegra element could not be created. Exiting.\n");
    return -1;
  }
#endif

  g_object_set (G_OBJECT (streammux), "width", MUXER_OUTPUT_WIDTH, "height",
      MUXER_OUTPUT_HEIGHT, "batch-size", 1, "live-source", 1,
      "batched-push-timeout", MUXER_BATCH_TIMEOUT_USEC, NULL);

  /* Set all the necessary properties of the nvinfer element,
   * the necessary ones are : */
  g_object_set (G_OBJECT (pgie),
      "config-file-path", "dstest1_pgie_config.txt", NULL);

  g_object_set (G_OBJECT (source), "device", "/dev/video1", "num-buffers", 900, NULL);

  caps_nv12_nvmm = gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING, "NV12", NULL);
  feature = gst_caps_features_new ("memory:NVMM", NULL);
  gst_caps_set_features (caps_nv12_nvmm, 0, feature);
  caps_nv12 = gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING, "NV12",
      "width", G_TYPE_INT, 640, "height", G_TYPE_INT,
      480, "framerate", GST_TYPE_FRACTION,
      30, 1, NULL);
  /* we add a message handler */
  bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
  bus_watch_id = gst_bus_add_watch (bus, bus_call, loop);
  gst_object_unref (bus);

  /* Set up the pipeline */
  /* we add all elements into the pipeline */
  gst_bin_add_many (GST_BIN (pipeline),
      source, conv1, nvconv, streammux, pgie,
      nvvidconv, nvosd, transform, sink, NULL); .

#ifdef NVVIDCONV
  gst_bin_add (GST_BIN (pipeline), conv2);
  gst_element_link_filtered(conv1, conv2, caps_nv12_nvmm);
  gst_element_link_filtered(conv2, nvconv, caps_nv12);
#else
  gst_element_link_filtered(conv1, nvconv, caps_nv12);
#endif
  gst_element_link_pads_filtered(nvconv, "src", streammux, "sink_0", caps_nv12_nvmm);

  gst_caps_unref(caps_nv12_nvmm);
  gst_caps_unref(caps_nv12);

  if (!gst_element_link_many (streammux, pgie,
      nvvidconv, nvosd, transform, sink, NULL)) {
    g_printerr ("Elements could not be linked: 2. Exiting.\n");
    return -1;
  }

  /* Lets add probe to get informed of the meta data generated, we add probe to
   * the sink pad of the osd element, since by that time, the buffer would have
   * had got all the metadata. */
  osd_sink_pad = gst_element_get_static_pad (nvosd, "sink");
  if (!osd_sink_pad)
    g_print ("Unable to get sink pad\n");
  else
    gst_pad_add_probe (osd_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
        osd_sink_pad_buffer_probe, NULL, NULL);

  /* Set the pipeline to "playing" state */
  gst_element_set_state (pipeline, GST_STATE_PLAYING);

  /* Wait till pipeline encounters an error or EOS */
  g_print ("Running...\n");
  g_main_loop_run (loop);

  /* Out of the main loop, clean up nicely */
  g_print ("Returned, stopping playback\n");
  gst_element_set_state (pipeline, GST_STATE_NULL);
  g_print ("Deleting pipeline\n");
  gst_object_unref (GST_OBJECT (pipeline));
  g_source_remove (bus_watch_id);
  g_main_loop_unref (loop);
  return 0;
}

However, I got the similar error with previous test on the terminal.

0:00:04.390426720 17069   0x5578e33200 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
0:00:04.394768704 17069   0x5578e33200 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:dstest1_pgie_config.txt sucessfully
Running...
ERROR from element v4l2-source: Internal data stream error.
Error details: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:dstest1-pipeline/GstV4l2Src:v4l2-source:
streaming stopped, reason not-negotiated (-4)
Returned, stopping playback
Deleting pipeline

Do you know how to solve this problem?
Any suggestion is appreciated, thanks.

Hi,
The format is not supported in v4l2src, so one possible solution is to use appsrc element and implement capturing frames and format conversion in appsrc. You may refer to the duscussion:
Jetson nano rtsp stream failed with intel realsense 415 usb camera - #15 by DaneLLL
How Deepstream reads depth data(Z16) and sends it via rtsp - #6 by DaneLLL

From DS5.0, there is a sample demonstrating appsrc. Please look at

/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-appsrc-test

Hi, @DaneLLL
The deepstream-appsrc-test example seems like use for video file input?
So what I need to do is modify video input to usb camera input?
And another question is that because v4l2src doesn’t support Z16, so I need to read data from cpu buffer instead of use gst_element_factory_make(“v4l2src”, “v4l2src”) to get stream data directly.
Is that correct?
Thanks.

Hi,
The sample reads video frames from file. Please refer to README for detail. And you would need to customize read_data() to capture Z16 frames and convert to I420/NV12/RGBA.

Since Z16 is a special format, it requires significant customization.

Hi, @DaneLLL,
Thanks for your reply, but I still confuse that in deepstream-appsrc-test example,
it use fopen to access the video frame data.

However, in my case, I don’t know how to access usb camera data by customize read_data() function.
Is there have some example or more detail document to understand how to access the usb camera data?
Really need your help, thanks.

Hi,
There are public samples. Please refer to

https://git.ideasonboard.org/yavta.git

Hi,
You can get the buffer pointer by calling:

  GstBuffer *buffer;
  GstMapInfo map;
  buffer = gst_buffer_new_allocate (NULL, data->frame_size, NULL);
  gst_buffer_map (buffer, &map, GST_MAP_WRITE);

The pointer is map.data. But not sure how to convertZ16 to I420(or NV12, RGBA). Need other users to share experience.