using nvivafilter customer-lib for C++ appsink

Hi, I’m developing with multiple cameras on Jetson Xavier.
Using nvivafilter I could manage to warp the images from camera on Gstreamer.
so the command line worked fine. (1.)
However when I tried to attach this on my program. it didn’t work.
I’ve attached code below, please refer (2., 4.)

  1. warped input to screen on command line (works)
gst-launch-1.0 -v pylonsrc camera=1 fps=30 ! bayer2rgb ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1920, height=1200, format=I420, framerate=30/1' ! nvivafilter customer-lib-name=./lib-gst-custom-opencv_cudaprocess1.so cuda-process=true ! 'video/x-raw(memory:NVMM), format=RGBA' ! nvegltransform ! nveglglessink
  1. warped input to appsink (Problem)
descr_string_vec[j] = "pylonsrc camera=" + std::to_string(j) + " fps=" + fps_s + " "
                             "! bayer2rgb "
                             "! nvvidconv top=0 bottom=1200 left=0 right=1920 "
                             "! video/x-raw(memory:NVMM), width=1920, height=1200, format=I420, framerate=(fraction)" + fps_s + "/1 "
                             "! nvivafilter customer-lib-name=../cu/lib-gst-custom-opencv_cudaprocess" + std::to_string(j) + ".so cuda-process=true "
                             "! video/x-raw(memory:NVMM), format=RGBA "
			     "! nvvidconv top=0 bottom=1200 left=0 right=1920 " 
			     "! video/x-raw, width=1920, height=1200, format=RGBA, framerate=(fraction)" + fps_s + "/1 "
                             "! appsink name=sink" + std::to_string(j) + " ";
  1. original input to appsink (works)
descr_string_vec[j] = "pylonsrc camera=" + std::to_string(j) + " fps=" + fps_s + " "
                             "! bayer2rgb "
                             "! videoconvert "
                             "! video/x-raw, format=BGR, framerate=(fraction)" + fps_s + "/1 "
                             "! appsink name=sink sync=true ";
  1. image gabbing code
GstFlowReturn
new_sample(GstAppSink *appsink, gpointer data) {
    static int framecount = 0;
    framecount++;

    GstSample *sample = gst_app_sink_pull_sample(appsink);
    GstCaps *caps = gst_sample_get_caps(sample);
    GstBuffer *buffer = gst_sample_get_buffer(sample);
    const GstStructure *info = gst_sample_get_info(sample);

    // ---- Read frame and convert to opencv format ---------------

    GstMapInfo map;
    gst_buffer_map (buffer, &map, GST_MAP_READ);

    // convert gstreamer data to OpenCV Mat, you could actually
    // resolve height / width from caps...

    ///how to directly gpumat
    //Mat frame(input_size, CV_8UC3, (char*)map.data, Mat::AUTO_STEP);
    cv::Mat frame(input_size, CV_8UC4, (char*)map.data, Mat::AUTO_STEP);

    // TODO: synchronize this....
    frameQueue[0].push_back(frame);

    gst_buffer_unmap(buffer, &map);

    // ------------------------------------------------------------

    // print dot every 30 frames
    if (framecount%30 == 0) {
        //g_print ("......");
    }

    // show caps on first frame
    if (framecount == 1) {
        g_print ("%s\n", gst_caps_to_string(caps));
    }

    gst_sample_unref (sample);
    return GST_FLOW_OK;
}

Q1) the result image of 2. is blue-ish warped image and red-ish small original images… (2.jpg)
Why it does not contain only warped image?

Q2) I’m trying to search as many as possible. however, the code 2. looks little bit inefficient.
Can I minimize the lines? is there any redundant usage?

I’ve attatched opencv conversion code 4. and images regarding code above for more clear understand.



I think that nvvidconv can operate in different modes. In standard way it would convert color format and rescale, as in your first pipeline, converting from RGB to I420 while resizing your camera resolution into 1920x1200.

In your second case pipeline, you are using ROI mode of nvvidconv, specifying a ROI of 1920x1200 from origin, although your camera has provided a smaller resoultion image. So the output frame will be just extended, not a rescale of input image. Your second nvvidconv crops the whole image while keeping the same RGBA format, so it does nothing.

You would decide what would be the resolution and format at each stage of the pipeline and check what you get, and finally just change sink. Get it first working with BGR using videoconvert, you may try RGBA to gtsreamer appsink then.

Additional note: I have no knowledge of basler cameras, but if your camera and pylonsrc can provide other formats than bayer, you may give it a try, debayering on cpu can be a significant load for high resolutions.
Furthermore, you may try to add queue plugins between elements of your pipeline, so that they can run on different CPU cores.