Latency at getting video frames via appsink from camera

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Hello,

i am trying to get a gstreamer chain up and running on the TX1 to get camera data as OpenCv::Mat. I started with a simple pipeline to capture the data from the camera and to display it on the display.

gst-launch-1.0 -e -v nvcamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! xvimagesink

Everything fine so far, no measurable latency.

Now when i exchange the xvimagesink with an appsink module to get a hand on the data, a delay of nearly 2 seconds is getting introduced.

It is not just a simple exchange, i build up the pipeline in source, translated as:

nvcamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)I420, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw, format=(string)I420' ! videoconvert ! 'video/x-raw, format=(string)BGR' ! appsink

and pull data cyclically via

bool VideoCaptureGst::getFrame
(
    cv::Mat& img,
    int camId
)
{
    bool gotFrame = false;

    // pull_sample will block until sample is ready. In our case "sample" is an encoded picture.
    GstSample* sample;
    sample = gst_app_sink_pull_sample(GST_APP_SINK(gstCamMaster.appsink));

    if(sample == NULL)
    {
        fprintf(stderr, "gst_app_sink_pull_sample returned null\n");
    } else {
        GstCaps *caps = gst_sample_get_caps(sample);

        GstStructure* s = gst_caps_get_structure (caps, 0);
        gint width, height;
        (void)gst_structure_get_int (s, "width", &width);
        (void)gst_structure_get_int (s, "height", &height);

        GstBuffer* buffer = gst_sample_get_buffer (sample);
        GstMapInfo map;
        gst_buffer_map (buffer, &map, GST_MAP_READ);

        // We do a deep copy of the data to pass it back to the caller
        cv::Mat imgbuf = cv::Mat(cv::Size(width, height), CV_8UC3, (char*)map.data, cv::Mat::AUTO_STEP);
        imgbuf.copyTo(img);

        gst_buffer_unmap (buffer, &map);
        gst_sample_unref (sample);

        gotFrame = true;
    }

    return gotFrame;
}

I already played arround with settings for the appsink which has no effect on the latency.

gst_app_sink_set_max_buffers(GST_APP_SINK(gstCamMaster.appsink), 1);
gst_app_sink_set_drop(GST_APP_SINK(gstCamMaster.appsink), TRUE);
g_object_set (G_OBJECT(gstCamMaster.appsink), "async", FALSE, "sync", FALSE, NULL);

Does anyone has an idea or can explain what is going on?

Thanks

2

Hi,
videoconvert may not have good performance. For OpenCv + gstreamer, please refer to
[url]https://devtalk.nvidia.com/default/topic/1024245/jetson-tx2/opencv-3-3-and-integrated-camera-problems-/post/5210735/#5210735[/url]