I am unable to process the video stream created with Gstreamer using OpenCV for face detection. (Jetson Nano Dev Kit)

ziyaelri · December 24, 2023, 1:50pm

Hi,
I want to create a project for face detection by processing the video stream I generate with Gstreamer using OpenCV. I am using a Jetson Nano with the ımx219-77 camera. Initially, I couldn’t even display the image on the screen, but I overcame this issue, and now I can display live video. However, at this stage, although I can display the image on the screen, I realized that I cannot process it with OpenCV. In Gstreamer, there is a function called ‘g_signal_connect,’ and I am using it, but I do not receive any return from this function every time a new frame arrives. Below, I will share the code, the output of the execution, and the ‘GST_DEBUG=3’ output. If a more detailed debug output is needed, just let me know.

Thank you again for your assistance. I am new to Gstreamer, so please enlighten me. Also, if I have misaddressed or omitted anything in the topic, I apologize to the moderators; please help me :).

It is my code;

#include <gst/gst.h>
#include <gst/app/gstappsink.h>
#include <opencv2/opencv.hpp>

GMainLoop *loop;

static gboolean bus_call(GstBus *bus, GstMessage *msg, gpointer data) {
    switch (GST_MESSAGE_TYPE(msg)) {
        case GST_MESSAGE_EOS:
            g_print("End of stream\n");
            g_main_loop_quit(loop);
            break;
        case GST_MESSAGE_ERROR: {
            gchar *debug;
            GError *error;

            gst_message_parse_error(msg, &error, &debug);
            g_free(debug);

            g_printerr("Error: %s\n", error->message);
            g_error_free(error);

            g_main_loop_quit(loop);
            break;
        }
        default:
            break;
    }

    return TRUE;
}

static  GstFlowReturn on_new_sample(GstElement *sink, gpointer user_data) {

    static int frame_count = 0;
    printf("Frame %d received\n", frame_count++);

    GstSample *sample = gst_app_sink_pull_sample(GST_APP_SINK(sink));
    g_print("on_new_sample called\n");

    if (sample) {
        GstBuffer *buffer = gst_sample_get_buffer(sample);
        GstMapInfo map;
        gst_buffer_map(buffer, &map, GST_MAP_READ);

        // Convert NV12 format to BGR format for OpenCV display
        cv::Mat frame_nv12(2464 + 2464 / 2, 3280, CV_8UC1, map.data);
        cv::Mat frame;
        cv::cvtColor(frame_nv12, frame, cv::COLOR_YUV2BGR_NV12);

        // Perform face detection using OpenCV Haar Cascade
        cv::CascadeClassifier face_cascade;
        if (!face_cascade.load("/usr/local/share/opencv4/haarcascades/haarcascade_frontalface_default.xml")) {
            std::cerr << "Haar Cascade dosyası yüklenemedi." << std::endl;
            //return GST_FLOW_ERROR;
        }

        std::vector<cv::Rect> faces;
        face_cascade.detectMultiScale(frame, faces, 1.1, 2, 0 | cv::CASCADE_SCALE_IMAGE, cv::Size(30, 30));

        // Draw rectangles around detected faces
        for (const auto &face : faces) {
            cv::rectangle(frame, face, cv::Scalar(0, 255, 0), 2);
        }

        // Display the frame using OpenCV
        cv::imshow("GStreamer OpenCV Display", frame);

        // Release the buffer
        gst_buffer_unmap(buffer, &map);
        gst_sample_unref(sample);

        // Handle keyboard input for exiting the loop
        if (cv::waitKey(10) == 27) {
            g_main_loop_quit(loop);
        }
    }

    return GST_FLOW_OK;
}

int main(int argc, char *argv[]) {
    // Initialize GStreamer
    g_main_context_push_thread_default(NULL);
    gst_init(&argc, &argv);

    // Create elements
    GstElement *pipeline = gst_pipeline_new("mypipeline");
    GstElement *source = gst_element_factory_make("nvarguscamerasrc", "src");
    GstElement *capsfilter1 = gst_element_factory_make("capsfilter", "filter1");
    GstElement *nvvidconv1 = gst_element_factory_make("nvvidconv", "nvvidconv1");
    GstElement *nvegltransform = gst_element_factory_make("nvegltransform", "nvegltransform");
    GstElement *appsink = gst_element_factory_make("nveglglessink", "appsink");  // Change to appsink - nveglglessink

    // Check for element creation errors
    if (!pipeline || !source || !capsfilter1 || !nvvidconv1 || !nvegltransform || !appsink) {
        g_printerr("Not all elements could be created. Exiting.\n");
        return -1;
    }

    // Set element properties
    g_object_set(G_OBJECT(capsfilter1), "caps",
                 gst_caps_from_string("video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=120/1"), NULL);

    // Configure appsink for efficient data retrieval
    //g_object_set(appsink, "emit-signals", TRUE, "sync", FALSE, "max-buffers", 1, NULL);
    g_signal_connect(appsink, "sample", G_CALLBACK(on_new_sample), NULL);

    // Add elements to pipeline and link them
    gst_bin_add_many(GST_BIN(pipeline), source, capsfilter1, nvvidconv1, nvegltransform, appsink, NULL);
    gst_element_link_many(source, capsfilter1, nvvidconv1, nvegltransform, appsink, NULL);

    // Add bus watch to listen for events
    GstBus *bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
    gst_bus_add_watch(bus, (GstBusFunc)bus_call, loop);
    gst_object_unref(bus);

    // Set pipeline to PLAYING state
    GstStateChangeReturn ret = gst_element_set_state(pipeline, GST_STATE_PLAYING);
    if (ret == GST_STATE_CHANGE_FAILURE) {
        g_printerr("Unable to set the pipeline to the playing state. Exiting.\n");
        gst_object_unref(pipeline);
        return -1;
    }

    // Create GMainLoop for handling GStreamer events
    loop = g_main_loop_new(nullptr, FALSE);

    // Run the main loop
    g_main_loop_run(loop);

    // Clean up
    gst_element_set_state(pipeline, GST_STATE_NULL);
    gst_object_unref(pipeline);
    g_main_loop_unref(loop);

    return 0;
}

It is my output;

(QTVideoTracking:18709): GLib-GObject-WARNING **: 16:36:19.054: ../../../../gobject/gsignal.c:2524: signal 'sample' is invalid for instance '0x556afc7940' of type 'GstEglGlesSink'

Using winsys: x11
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0
   Camera mode  = 5
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 120.000005
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.

and It is my Gst_debug=3 output;



futuris@futuris-desktop:~/Desktop/gstreamerImx/QtVideoTrack/build-QTVideoTracking-Desktop-Debug$ GST_DEBUG=3 ./QTVideoTracking

(QTVideoTracking:19447): GLib-GObject-WARNING **: 16:41:44.842: ../../../../gobject/gsignal.c:2524: signal 'sample' is invalid for instance '0x55a782d940' of type 'GstEglGlesSink'

Using winsys: x11 
0:00:00.483651406 19447   0x55a78204a0 FIXME                default gstutils.c:3981:gst_pad_create_stream_id_internal:<src:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.

and It is GST_DEBUG=4 output;

futuris@futuris-desktop:~/Desktop/gstreamerImx/QtVideoTrack/build-QTVideoTracking-Desktop-Debug$ GST_DEBUG=4 ./QTVideoTracking
0:00:00.000159583 19782   0x559aa96200 INFO                GST_INIT gst.c:586:init_pre: Initializing GStreamer Core Library version 1.14.5
0:00:00.000225104 19782   0x559aa96200 INFO                GST_INIT gst.c:587:init_pre: Using library installed in /usr/lib/aarch64-linux-gnu
0:00:00.000260677 19782   0x559aa96200 INFO                GST_INIT gst.c:607:init_pre: Linux futuris-desktop 4.9.253-tegra #1 SMP PREEMPT Mon Jul 26 12:13:06 PDT 2021 aarch64
0:00:00.000769531 19782   0x559aa96200 INFO                GST_INIT gstmessage.c:127:_priv_gst_message_initialize: init messages
0:00:00.001678177 19782   0x559aa96200 INFO                GST_INIT gstcontext.c:84:_priv_gst_context_initialize: init contexts
0:00:00.002077812 19782   0x559aa96200 INFO      GST_PLUGIN_LOADING gstplugin.c:317:_priv_gst_plugin_initialize: registering 0 static plugins
0:00:00.002279791 19782   0x559aa96200 INFO      GST_PLUGIN_LOADING gstplugin.c:225:gst_plugin_register_static: registered static plugin "staticelements"
0:00:00.002308802 19782   0x559aa96200 INFO      GST_PLUGIN_LOADING gstplugin.c:227:gst_plugin_register_static: added static plugin "staticelements", result: 1
0:00:00.002356146 19782   0x559aa96200 INFO            GST_REGISTRY gstregistry.c:1727:ensure_current_registry: reading registry cache: /home/futuris/.cache/gstreamer-1.0/registry.aarch64.bin
0:00:00.048161302 19782   0x559aa96200 INFO            GST_REGISTRY gstregistrybinary.c:621:priv_gst_registry_binary_read_cache: loaded /home/futuris/.cache/gstreamer-1.0/registry.aarch64.bin in 0.045731 seconds
0:00:00.048308281 19782   0x559aa96200 INFO            GST_REGISTRY gstregistry.c:1583:scan_and_update_registry: Validating plugins from registry cache: /home/futuris/.cache/gstreamer-1.0/registry.aarch64.bin
0:00:00.055634323 19782   0x559aa96200 INFO            GST_REGISTRY gstregistry.c:1685:scan_and_update_registry: Registry cache has not changed
0:00:00.055686250 19782   0x559aa96200 INFO            GST_REGISTRY gstregistry.c:1762:ensure_current_registry: registry reading and updating done, result = 1
0:00:00.055711093 19782   0x559aa96200 INFO                GST_INIT gst.c:807:init_post: GLib runtime version: 2.56.4
0:00:00.055736145 19782   0x559aa96200 INFO                GST_INIT gst.c:809:init_post: GLib headers version: 2.56.4
0:00:00.055754583 19782   0x559aa96200 INFO                GST_INIT gst.c:810:init_post: initialized GStreamer successfully
0:00:00.055823593 19782   0x559aa96200 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "pipeline" named "mypipeline"
0:00:00.476712135 19782   0x559aa96200 INFO      GST_PLUGIN_LOADING gstplugin.c:901:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so" loaded
0:00:00.476764374 19782   0x559aa96200 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "nvarguscamerasrc" named "src"
0:00:00.477204999 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseSrc@0x559ad295d0> adding pad 'src'
0:00:00.478694010 19782   0x559aa96200 INFO      GST_PLUGIN_LOADING gstplugin.c:901:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstcoreelements.so" loaded
0:00:00.478737083 19782   0x559aa96200 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "capsfilter" named "filter1"
0:00:00.478894270 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseTransform@0x559ad36260> adding pad 'sink'
0:00:00.478941666 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseTransform@0x559ad36260> adding pad 'src'
0:00:00.479872552 19782   0x559aa96200 INFO      GST_PLUGIN_LOADING gstplugin.c:901:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so" loaded
0:00:00.479911145 19782   0x559aa96200 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "nvvidconv" named "nvvidconv1"
0:00:00.480146718 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseTransform@0x559abe84d0> adding pad 'sink'
0:00:00.480192604 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseTransform@0x559abe84d0> adding pad 'src'
0:00:00.481337916 19782   0x559aa96200 INFO      GST_PLUGIN_LOADING gstplugin.c:901:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvegltransform.so" loaded
0:00:00.481376562 19782   0x559aa96200 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "nvegltransform" named "nvegltransform"
0:00:00.481547031 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseTransform@0x559ad3e120> adding pad 'sink'
0:00:00.481590677 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseTransform@0x559ad3e120> adding pad 'src'
0:00:00.483932760 19782   0x559aa96200 INFO      GST_PLUGIN_LOADING gstplugin.c:901:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglglessink.so" loaded
0:00:00.483979583 19782   0x559aa96200 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element "nveglglessink" named "appsink"
0:00:00.484364531 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstelement.c:670:gst_element_add_pad:<GstBaseSink@0x559ad45750> adding pad 'sink'
0:00:00.484530468 19782   0x559aa96200 INFO               GST_EVENT gstevent.c:1517:gst_event_new_reconfigure: creating reconfigure event

(QTVideoTracking:19782): GLib-GObject-WARNING **: 16:45:06.138: ../../../../gobject/gsignal.c:2524: signal 'sample' is invalid for instance '0x559ad45750' of type 'GstEglGlesSink'
0:00:00.484819791 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstutils.c:1774:gst_element_link_pads_full: trying to link element src:(any) to element filter1:(any)
0:00:00.484854635 19782   0x559aa96200 INFO                GST_PADS gstutils.c:1035:gst_pad_check_link: trying to link src:src and filter1:sink
0:00:00.484906458 19782   0x559aa96200 INFO                GST_PADS gstpad.c:4232:gst_pad_peer_query:<filter1:src> pad has no peer
0:00:00.484955572 19782   0x559aa96200 INFO                GST_PADS gstutils.c:1588:prepare_link_maybe_ghosting: src and filter1 in same bin, no need for ghost pads
0:00:00.484997343 19782   0x559aa96200 INFO                GST_PADS gstpad.c:2378:gst_pad_link_prepare: trying to link src:src and filter1:sink
0:00:00.485029999 19782   0x559aa96200 INFO                GST_PADS gstpad.c:4232:gst_pad_peer_query:<filter1:src> pad has no peer
0:00:00.485067031 19782   0x559aa96200 INFO                GST_PADS gstpad.c:2586:gst_pad_link_full: linked src:src and filter1:sink, successful
0:00:00.485086562 19782   0x559aa96200 INFO               GST_EVENT gstevent.c:1517:gst_event_new_reconfigure: creating reconfigure event
0:00:00.485107552 19782   0x559aa96200 INFO               GST_EVENT gstpad.c:5808:gst_pad_send_event_unchecked:<src:src> Received event on flushing pad. Discarding
0:00:00.485147291 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstutils.c:1774:gst_element_link_pads_full: trying to link element filter1:(any) to element nvvidconv1:(any)
0:00:00.485175989 19782   0x559aa96200 INFO                GST_PADS gstutils.c:1035:gst_pad_check_link: trying to link filter1:src and nvvidconv1:sink
0:00:00.485221302 19782   0x559aa96200 INFO                GST_PADS gstpad.c:4232:gst_pad_peer_query:<nvvidconv1:src> pad has no peer
0:00:00.485407499 19782   0x559aa96200 INFO                GST_PADS gstutils.c:1588:prepare_link_maybe_ghosting: filter1 and nvvidconv1 in same bin, no need for ghost pads
0:00:00.485446614 19782   0x559aa96200 INFO                GST_PADS gstpad.c:2378:gst_pad_link_prepare: trying to link filter1:src and nvvidconv1:sink
0:00:00.485490885 19782   0x559aa96200 INFO                GST_PADS gstpad.c:4232:gst_pad_peer_query:<nvvidconv1:src> pad has no peer
0:00:00.485635156 19782   0x559aa96200 INFO                GST_PADS gstpad.c:2586:gst_pad_link_full: linked filter1:src and nvvidconv1:sink, successful
0:00:00.485656041 19782   0x559aa96200 INFO               GST_EVENT gstevent.c:1517:gst_event_new_reconfigure: creating reconfigure event
0:00:00.485676666 19782   0x559aa96200 INFO               GST_EVENT gstpad.c:5808:gst_pad_send_event_unchecked:<filter1:src> Received event on flushing pad. Discarding
0:00:00.485711927 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstutils.c:1774:gst_element_link_pads_full: trying to link element nvvidconv1:(any) to element nvegltransform:(any)
0:00:00.485741145 19782   0x559aa96200 INFO                GST_PADS gstutils.c:1035:gst_pad_check_link: trying to link nvvidconv1:src and nvegltransform:sink
0:00:00.485851458 19782   0x559aa96200 INFO                GST_PADS gstpad.c:4232:gst_pad_peer_query:<nvegltransform:src> pad has no peer
0:00:00.485973177 19782   0x559aa96200 INFO                GST_PADS gstutils.c:1588:prepare_link_maybe_ghosting: nvvidconv1 and nvegltransform in same bin, no need for ghost pads
0:00:00.486011822 19782   0x559aa96200 INFO                GST_PADS gstpad.c:2378:gst_pad_link_prepare: trying to link nvvidconv1:src and nvegltransform:sink
0:00:00.486124479 19782   0x559aa96200 INFO                GST_PADS gstpad.c:4232:gst_pad_peer_query:<nvegltransform:src> pad has no peer
0:00:00.486217708 19782   0x559aa96200 INFO                GST_PADS gstpad.c:2586:gst_pad_link_full: linked nvvidconv1:src and nvegltransform:sink, successful
0:00:00.486237604 19782   0x559aa96200 INFO               GST_EVENT gstevent.c:1517:gst_event_new_reconfigure: creating reconfigure event
0:00:00.486257604 19782   0x559aa96200 INFO               GST_EVENT gstpad.c:5808:gst_pad_send_event_unchecked:<nvvidconv1:src> Received event on flushing pad. Discarding
0:00:00.486291093 19782   0x559aa96200 INFO        GST_ELEMENT_PADS gstutils.c:1774:gst_element_link_pads_full: trying to link element nvegltransform:(any) to element appsink:(any)
0:00:00.486320052 19782   0x559aa96200 INFO                GST_PADS gstutils.c:1035:gst_pad_check_link: trying to link nvegltransform:src and appsink:sink
0:00:00.486553072 19782   0x559aa96200 INFO                GST_PADS gstutils.c:1588:prepare_link_maybe_ghosting: nvegltransform and appsink in same bin, no need for ghost pads
0:00:00.486596249 19782   0x559aa96200 INFO                GST_PADS gstpad.c:2378:gst_pad_link_prepare: trying to link nvegltransform:src and appsink:sink
0:00:00.486817552 19782   0x559aa96200 INFO                GST_PADS gstpad.c:2586:gst_pad_link_full: linked nvegltransform:src and appsink:sink, successful
0:00:00.486838385 19782   0x559aa96200 INFO               GST_EVENT gstevent.c:1517:gst_event_new_reconfigure: creating reconfigure event
0:00:00.486859999 19782   0x559aa96200 INFO               GST_EVENT gstpad.c:5808:gst_pad_send_event_unchecked:<nvegltransform:src> Received event on flushing pad. Discarding
0:00:00.486968281 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<appsink> current NULL pending VOID_PENDING, desired next READY

Using winsys: x11 
0:00:00.487179166 19782   0x559aa96200 INFO             egladaption gstegladaptation_egl.c:186:gst_egl_adaptation_init_display:<appsink> System reports supported EGL version v1.5
0:00:00.487550364 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<appsink> completed state change to READY
0:00:00.487590156 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<appsink> notifying about state-changed NULL to READY (VOID_PENDING pending)
0:00:00.487629635 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'appsink' changed state to 2(READY) successfully
0:00:00.487663489 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvegltransform> current NULL pending VOID_PENDING, desired next READY
0:00:00.487689583 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<nvegltransform> completed state change to READY
0:00:00.487714114 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvegltransform> notifying about state-changed NULL to READY (VOID_PENDING pending)
0:00:00.487743802 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'nvegltransform' changed state to 2(READY) successfully
0:00:00.487773072 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvvidconv1> current NULL pending VOID_PENDING, desired next READY
0:00:00.487798333 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<nvvidconv1> completed state change to READY
0:00:00.487821249 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvvidconv1> notifying about state-changed NULL to READY (VOID_PENDING pending)
0:00:00.487850416 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'nvvidconv1' changed state to 2(READY) successfully
0:00:00.487879270 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<filter1> current NULL pending VOID_PENDING, desired next READY
0:00:00.487902604 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<filter1> completed state change to READY
0:00:00.487924947 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<filter1> notifying about state-changed NULL to READY (VOID_PENDING pending)
0:00:00.487964583 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'filter1' changed state to 2(READY) successfully
0:00:00.487992239 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<src> current NULL pending VOID_PENDING, desired next READY
0:00:00.488015937 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<src> completed state change to READY
0:00:00.488038593 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<src> notifying about state-changed NULL to READY (VOID_PENDING pending)
0:00:00.488068437 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'src' changed state to 2(READY) successfully
0:00:00.488101093 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2651:gst_element_continue_state:<mypipeline> committing state from NULL to READY, pending PLAYING, next PAUSED
0:00:00.488124635 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<mypipeline> notifying about state-changed NULL to READY (PLAYING pending)
0:00:00.488154166 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2658:gst_element_continue_state:<mypipeline> continue state change READY to PAUSED, final PLAYING
0:00:00.488194791 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<appsink> current READY pending VOID_PENDING, desired next PAUSED
0:00:00.495979843 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2960:gst_bin_change_state_func:<mypipeline> child 'appsink' is changing state asynchronously to PAUSED
0:00:00.496043229 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvegltransform> current READY pending VOID_PENDING, desired next PAUSED
0:00:00.496240104 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<nvegltransform> completed state change to PAUSED
0:00:00.496282812 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvegltransform> notifying about state-changed READY to PAUSED (VOID_PENDING pending)
0:00:00.496323072 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'nvegltransform' changed state to 3(PAUSED) successfully
0:00:00.496358177 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvvidconv1> current READY pending VOID_PENDING, desired next PAUSED
0:00:00.496626249 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<nvvidconv1> completed state change to PAUSED
0:00:00.496659427 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvvidconv1> notifying about state-changed READY to PAUSED (VOID_PENDING pending)
0:00:00.496696041 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'nvvidconv1' changed state to 3(PAUSED) successfully
0:00:00.496730260 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<filter1> current READY pending VOID_PENDING, desired next PAUSED
0:00:00.496767239 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<filter1> completed state change to PAUSED
0:00:00.496790833 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<filter1> notifying about state-changed READY to PAUSED (VOID_PENDING pending)
0:00:00.496876302 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'filter1' changed state to 3(PAUSED) successfully
0:00:00.496908177 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<src> current READY pending VOID_PENDING, desired next PAUSED
0:00:00.497014583 19782   0x559aa96200 INFO                 basesrc gstbasesrc.c:1358:gst_base_src_do_seek:<src> seeking: time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.000000000, duration 99:99:99.999999999
0:00:00.497122812 19782   0x559aa96200 INFO                    task gsttask.c:457:gst_task_set_lock: setting stream lock 0x559ad30130 on task 0x559ad97050
0:00:00.497149270 19782   0x559aa96200 INFO                GST_PADS gstpad.c:6154:gst_pad_start_task:<src:src> created task 0x559ad97050
0:00:00.497403697 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<src> completed state change to PAUSED
0:00:00.497449531 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<src> notifying about state-changed READY to PAUSED (VOID_PENDING pending)
0:00:00.497518124 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2997:gst_bin_change_state_func:<mypipeline> child 'src' changed state to 3(PAUSED) successfully without preroll
0:00:00.497588541 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2651:gst_element_continue_state:<mypipeline> committing state from READY to PAUSED, pending PLAYING, next PLAYING
0:00:00.497613697 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<mypipeline> notifying about state-changed READY to PAUSED (PLAYING pending)
0:00:00.497677343 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2658:gst_element_continue_state:<mypipeline> continue state change PAUSED to PLAYING, final PLAYING
0:00:00.497917291 19782   0x559aa96200 INFO               GST_EVENT gstevent.c:1388:gst_event_new_latency: creating latency event 0:00:00.000000000
0:00:00.498003645 19782   0x559aa96200 INFO                     bin gstbin.c:2783:gst_bin_do_latency_func:<mypipeline> configured latency of 0:00:00.000000000
0:00:00.498060052 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<appsink> current READY pending PAUSED, desired next PLAYING
0:00:00.498088489 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2960:gst_bin_change_state_func:<mypipeline> child 'appsink' is changing state asynchronously to PLAYING
0:00:00.498119322 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvegltransform> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:00.498145052 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<nvegltransform> completed state change to PLAYING
0:00:00.498169322 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvegltransform> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:00.498198437 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'nvegltransform' changed state to 4(PLAYING) successfully
0:00:00.498230052 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvvidconv1> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:00.498256666 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<nvvidconv1> completed state change to PLAYING
0:00:00.498282031 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvvidconv1> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:00.498313489 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'nvvidconv1' changed state to 4(PLAYING) successfully
0:00:00.498446458 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<filter1> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:00.498476458 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<filter1> completed state change to PLAYING
0:00:00.498501354 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<filter1> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:00.498538177 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'filter1' changed state to 4(PLAYING) successfully
0:00:00.498569999 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<src> completed state change to PLAYING
0:00:00.498594687 19782   0x559aa96200 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<src> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:00.498626614 19782   0x559aa96200 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'src' changed state to 4(PLAYING) successfully
0:00:00.499093124 19782   0x559ad384a0 INFO        GST_ELEMENT_PADS gstelement.c:917:gst_element_get_static_pad: no such pad 'sink' in element "src"
0:00:00.499145677 19782   0x559ad384a0 FIXME                default gstutils.c:3981:gst_pad_create_stream_id_internal:<src:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
0:00:00.500356041 19782   0x559ad384a0 INFO               GST_EVENT gstevent.c:814:gst_event_new_caps: creating caps event video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
0:00:00.500483229 19782   0x559ad384a0 INFO           basetransform gstbasetransform.c:1308:gst_base_transform_setcaps:<filter1> reuse caps
0:00:00.500543749 19782   0x559ad384a0 INFO               GST_EVENT gstevent.c:814:gst_event_new_caps: creating caps event video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
0:00:00.500958229 19782   0x559ad384a0 INFO           basetransform gstbasetransform.c:1308:gst_base_transform_setcaps:<nvvidconv1> reuse caps
0:00:00.501023072 19782   0x559ad384a0 INFO               GST_EVENT gstevent.c:814:gst_event_new_caps: creating caps event video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1
0:00:00.501163906 19782   0x559ad384a0 INFO               GST_EVENT gstevent.c:814:gst_event_new_caps: creating caps event video/x-raw(memory:EGLImage), width=(int)1280, height=(int)720, format=(string)RGBA, framerate=(fraction)120/1
0:00:00.507372552 19782   0x559ad380f0 INFO           nveglglessink gsteglglessink.c:2685:gst_eglglessink_configure_caps:<appsink> No window. Will attempt internal window creation
0:00:00.507424687 19782   0x559ad380f0 INFO           nveglglessink gsteglglessink.c:937:gst_eglglessink_create_window:<appsink> Attempting internal window creation
GST_ARGUS: Creating output stream
0:00:00.623783020 19782   0x559ad380f0 INFO             egladaption gstegladaptation_egl.c:254:gst_egl_adaptation_update_surface_dimensions:<appsink> Got surface of 1280x720 pixels
0:00:00.627379427 19782   0x559ad380f0 INFO             egladaption gstegladaptation.c:611:gst_egl_adaptation_init_surface:<appsink> Performing initial texture setup
0:00:00.627524791 19782   0x559ad380f0 INFO           nveglglessink gsteglglessink.c:2726:gst_eglglessink_configure_caps:<appsink> Configured caps successfully
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
0:00:00.858522187 19782   0x559ad384a0 INFO               GST_EVENT gstevent.c:895:gst_event_new_segment: creating segment event time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.000000000, duration 99:99:99.999999999
0:00:00.858631874 19782   0x559ad384a0 INFO                 basesrc gstbasesrc.c:2945:gst_base_src_loop:<src> marking pending DISCONT
0:00:00.891259531 19782   0x559ad380f0 INFO           nveglglessink gsteglglessink.c:980:gst_eglglessink_setup_vbo:<appsink> VBO setup. have_vbo:0
0:00:00.901362343 19782   0x559ad384a0 INFO              GST_STATES gstbin.c:3433:bin_handle_async_done:<mypipeline> setting state from PAUSED to PAUSED, pending PLAYING
0:00:00.901410468 19782   0x559ad384a0 INFO              GST_STATES gstbin.c:3453:bin_handle_async_done:<mypipeline> continue state change, pending PLAYING
0:00:00.901679010 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:3250:gst_bin_continue_func:<mypipeline> continue state change PAUSED to PLAYING, final PLAYING
0:00:00.901843697 19782   0x7f8c004680 INFO               GST_EVENT gstevent.c:1388:gst_event_new_latency: creating latency event 0:00:00.000000000
0:00:00.901926718 19782   0x7f8c004680 INFO                     bin gstbin.c:2783:gst_bin_do_latency_func:<mypipeline> configured latency of 0:00:00.000000000
0:00:00.902041145 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<appsink> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:00.902088489 19782   0x7f8c004680 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<appsink> completed state change to PLAYING
0:00:00.902115781 19782   0x7f8c004680 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<appsink> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:00.902247603 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'appsink' changed state to 4(PLAYING) successfully
0:00:00.902296926 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvegltransform> current PLAYING pending VOID_PENDING, desired next PLAYING
0:00:00.902330051 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2623:gst_bin_element_set_state:<nvegltransform> skipping transition from PLAYING to  PLAYING
0:00:00.902362708 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'nvegltransform' changed state to 4(PLAYING) successfully
0:00:00.902400416 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvvidconv1> current PLAYING pending VOID_PENDING, desired next PLAYING
0:00:00.902441197 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2623:gst_bin_element_set_state:<nvvidconv1> skipping transition from PLAYING to  PLAYING
0:00:00.902466301 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'nvvidconv1' changed state to 4(PLAYING) successfully
0:00:00.902494322 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<filter1> current PLAYING pending VOID_PENDING, desired next PLAYING
0:00:00.902535937 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2623:gst_bin_element_set_state:<filter1> skipping transition from PLAYING to  PLAYING
0:00:00.902563385 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'filter1' changed state to 4(PLAYING) successfully
0:00:00.903116614 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2506:gst_bin_element_set_state:<src> current PLAYING pending VOID_PENDING, desired next PLAYING
0:00:00.903169635 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2623:gst_bin_element_set_state:<src> skipping transition from PLAYING to  PLAYING
0:00:00.903200468 19782   0x7f8c004680 INFO              GST_STATES gstbin.c:2954:gst_bin_change_state_func:<mypipeline> child 'src' changed state to 4(PLAYING) successfully
0:00:00.903241770 19782   0x7f8c004680 INFO              GST_STATES gstelement.c:2676:gst_element_continue_state:<mypipeline> completed state change to PLAYING
0:00:00.903286458 19782   0x7f8c004680 INFO              GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<mypipeline> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)

I thank you again. If GST_DEBUG=5 is needed, I can share that as well. And I emphasize once again, if I have done anything wrong at any point, just let me know, and I will correct it. :) Wishing you a productive day.

rajupadhyay59 · December 25, 2023, 12:07am

The pipeline seems wrong.

Also, this is wrong
GstElement *appsink = gst_element_factory_make("nveglglessink", "appsink");

Change it to
GstElement *appsink = gst_element_factory_make("appsink", "appsink");

Remove nvegltransform before the appsink,
Also, to access the buffer, you need to set_property in appsink emit-signals=true since it is false by default.
Maybe set sync=false for now. (appsink set_property)

Refer to some cpp or even python gstreamer code to get an idea of how the pipeline works.
pipeline would be something like this
source -> caps -> nvvidconv -> appsink

Refer to this code for reference.

ziyaelri · December 25, 2023, 7:48am

Good morning,

Thanks for your response. I’ve fixed my pipeline, and now I can reach the ‘on_new_sample’ function. To match the frames of GStreamer and OpenCV, I modified the ‘on_new_sample’ function as follows.

static  GstFlowReturn on_new_sample(GstElement *sink, gpointer user_data) {

    static int frame_count = 0;
    printf("Frame %d received\n", frame_count++);

    GstSample *sample = gst_app_sink_pull_sample(GST_APP_SINK(sink));
    g_print("on_new_sample called\n");

    if (sample) {
        GstBuffer *buffer = gst_sample_get_buffer(sample);
        GstMapInfo map;
        gst_buffer_map(buffer, &map, GST_MAP_READ);

        // Convert NV12 format to BGR format for OpenCV display
        cv::Mat frame_nv12(480 * 3 / 2, 1280, CV_8UC1, map.data);

//        cv::Mat frame;
//        cv::cvtColor(frame_nv12, frame, cv::COLOR_YUV2BGR_NV12);

        // Perform face detection using OpenCV Haar Cascade
        cv::CascadeClassifier face_cascade;
        if (!face_cascade.load("/usr/local/share/opencv4/haarcascades/haarcascade_frontalface_default.xml")) {
            std::cerr << "Haar Cascade dosyası yüklenemedi." << std::endl;
            //return GST_FLOW_ERROR;
        }

//        std::vector<cv::Rect> faces;
//        face_cascade.detectMultiScale(frame_nv12, faces, 1.1, 2, 0 | cv::CASCADE_SCALE_IMAGE, cv::Size(30, 30));

//        // Draw rectangles around detected faces
//        for (const auto &face : faces) {
//            cv::rectangle(frame_nv12, face, cv::Scalar(0, 255, 0), 2);
//        }

        // Display the frame using OpenCV
        cv::imshow("GStreamer OpenCV Display", frame_nv12);

        // Release the buffer
        gst_buffer_unmap(buffer, &map);
        gst_sample_unref(sample);

        // Handle keyboard input for exiting the loop
        if (cv::waitKey(10) == 27) {
            g_main_loop_quit(loop);
        }
    }

    return GST_FLOW_OK;
}

However, after starting the code, it remains as follows. As far as I understand, either new frames are not coming, or I cannot convert it to a format supported by OpenCV from NV12. Do you have any advice on this situation? Thanks again. :) Have a good day.


GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Frame 0 received
on_new_sample called

rajupadhyay59 · December 25, 2023, 9:09am

Can you then try using RGBA instead of NV12 just for the sake of it?

GstFlowReturn on_new_sample(GstElement *appsink, gpointer udata) {
    GstSample *sample = gst_app_sink_pull_sample(GST_APP_SINK(appsink));
    if (!sample) {
        return GST_FLOW_ERROR;
    }

    GstBuffer *inbuf = gst_sample_get_buffer(sample);
    if (!inbuf) {
        gst_sample_unref(sample);
        g_printerr("error while retrieving buffer from sample\n");
        return GST_FLOW_ERROR;
    }

    cv::cuda::setDevice(0);

    GstMapInfo in_map_info;
    memset (&in_map_info, 0, sizeof(in_map_info));

    if (!gst_buffer_map (inbuf, &in_map_info, GST_MAP_READ)) {
        g_print ("Error: Failed to map gst buffer\n");
        return GST_FLOW_ERROR;
    }

    NvBufSurface* surface = NULL;
    surface = (NvBufSurface *) in_map_info.data;

    if (USE_EGLIMAGE) {
        if (NvBufSurfaceMapEglImage (surface, 0) !=0 ) {
            g_printerr("NvBufSurfaceMapEglImage error\n");
            return GST_FLOW_ERROR;
        }

        static bool create_filter = true;
        static cv::Ptr< cv::cuda::Filter > filter;
        CUresult status;
        CUeglFrame eglFrame;
        CUgraphicsResource pResource = NULL;
        cudaFree(0);
        status = cuGraphicsEGLRegisterImage(&pResource,
                                            surface->surfaceList[0].mappedAddr.eglImage,
                                            CU_GRAPHICS_MAP_RESOURCE_FLAGS_NONE);
        status = cuGraphicsResourceGetMappedEglFrame(&eglFrame, pResource, 0, 0);
        status = cuCtxSynchronize();

        /*if (create_filter) {
            filter = cv::cuda::createSobelFilter(CV_8UC4, CV_8UC4, 1, 0, 3, 1, cv::BORDER_DEFAULT);
            //filter = cv::cuda::createGaussianFilter(CV_8UC4, CV_8UC4, cv::Size(31,31), 0, 0, cv::BORDER_DEFAULT);
            create_filter = false;
        }*/

        cv::cuda::GpuMat d_mat(1344, 1344, CV_8UC4, eglFrame.frame.pPitch[0]);

        /*cv::Mat a;
        d_mat.download(a);
        cv::imwrite("a.jpg", a);*/


        // filter->apply (d_mat, d_mat);//
        status = cuCtxSynchronize();
        status = cuGraphicsUnregisterResource(pResource);
        // NvBufSurfTransform (dsexample->inter_buf, &ip_surf, &transform_params);
        // Destroy the EGLImage
        NvBufSurfaceUnMapEglImage (surface, 0);
        gst_buffer_unmap(inbuf, &in_map_info);
    }

    gst_sample_unref (sample);

    return GST_FLOW_OK;
}

It is a bit messy but you can refer this.
I cannot send the entire code because I cannot share some parts but even in my pipeline, appsink was following nvvidconv so you should be fine.

Another thing is that I mapped the nvmm:memory to eglimage to get gpumat directly instead of cpumat (taking reference from a code I found in nvidia developer forum).

Replace the snippet to get cpumat instead of gpumat. Again, try it with RGBA once.

ziyaelri · December 25, 2023, 12:14pm

Hello, I tried but it didn’t work, so I went back to the beginning. Now, I believe I’m converting NV12 to a format supported by OpenCV, but the program crashes inside the ‘imshow()’ function. I really don’t understand what’s going wrong.

rajupadhyay59 · December 26, 2023, 12:20am

try saving the image rather than displaying it.

ziyaelri · December 28, 2023, 7:25am

Good morning,

I am grateful for everything and thank you. I believe I have solved the image format issue (it can be done with ‘videoconvert’ in the GStreamer pipeline). However, I have another question to ask. The issue I am currently facing is ‘max buffers reached.’ 'g_signal_connect()’ is working correctly, and I can reach the part where I process the image with OpenCV. However, it seems that frames are coming too fast, causing the buffer to overflow. As a result, the code gets stuck. To overcome this problem, I am trying to use a mutex. I have created two separate threads. One is used to store frames in a matrix (like ‘cv::Mat frame1’), and the other is used to retrieve this matrix as an image or video. Do you have any advice or suggestions on how to solve this issue? I apologize for any inconvenience I may have caused and thank you again. Wishing you a productive day :)

rajupadhyay59 · December 28, 2023, 7:35am

Weird, I remember being able to do it with nvvidconv but I guess videoconvert is alright too since you are using appsink anyways.

I am not sure what you mean by the other stuff but if your frames are coming faster than your processing stuff, then you can use something known as ‘videorate’ element in gstreamer.

e.g., ... ! videorate ! video/x-raw(memory:NVMM,framerate=15/1) ! ...
Doing this, you will receive the frames at the rate of 15FPS.

If you want to drop frames or something, you can introduce queue to your pipeline and do the following.
... ! queue max-size-buffers=1 leaky=2 ! ...

gst-inspect-1.0 queue to know more about the properties.

Thing about mutex is, I am unsure how effective it will be because while one is saving, that time the file is locked right? So it cannot be received.
When one receives it, that the time file cannot be written.
So it will not be as effective. I would personally recommend using videorate to set your framerate.
You’ll find a lot of examples in this forum itself.

ziyaelri · January 1, 2024, 3:00pm

Hello,

How are you? I tried your suggestions and made progress. However, I encountered a point where I got stuck, and as far as I understand, it’s related to OpenCV. Now, I can capture frames and pass them to OpenCV for processing. But when I try to display them on the screen (using imshow), a window is created with a black interior and immediately closes. As far as I understand, this issue is related to ‘cv::waitKey(int i)’.

To test this, I removed ‘cv::waitKey’ from the code (using imshow and imwrite functions) and checked the state of the image. Both when saving as an image and when providing it as a video on the screen, a constant and unchanging snowy, corrupted image appeared, and the console continuously printed output indicating that frames were being received.

When I didn’t remove the ‘cv::waitKey’ function, in both saving as an image and providing it as a video, it saved/played a single frame (a black image), and then the code stopped working with a ‘segmentation fault’.

It seems that when I set the “esc” or “q” keys to exit from cv::waitKey , and I start the code with the intention of closing the video using these keys, the keys do not respond at all.

Below, I’ll share the code and GStreamer debug output in order. Do you have any advice you could provide? Thanks for your help.

#include <gst/gst.h>
#include <gst/app/gstappsink.h>
#include <opencv2/opencv.hpp>
#include <thread>
#include <mutex>
#include <X11/Xlib.h>


GMainLoop *loop = nullptr;
static int frame_counter = 0;
static int frameCounter = 0;
cv::Mat next_frame=cv::Mat::zeros(1280, 720, CV_8UC3);

std::mutex frameMutex;




static gboolean bus_call(GstBus *bus, GstMessage *msg, gpointer data) {
    switch (GST_MESSAGE_TYPE(msg)) {
        case GST_MESSAGE_EOS:
            g_print("End of stream\n");
            g_main_loop_quit(loop);
            break;
        case GST_MESSAGE_ERROR: {
            gchar *debug;
            GError *error;

            gst_message_parse_error(msg, &error, &debug);
            g_free(debug);

            g_printerr("Error: %s\n", error->message);
            g_error_free(error);

            g_main_loop_quit(loop);
            break;
        }
        default:
            break;
    }

    return TRUE;
}

static GstFlowReturn on_new_sample(GstElement *sink, gpointer user_data) {
    frame_counter++;  // Her çağrıldığında sayaç değerini artır

    GstSample *sample = gst_app_sink_pull_sample(GST_APP_SINK(sink));
    if (sample) {
        
        GstCaps *caps = gst_sample_get_caps(sample);
        GstStructure *structure = gst_caps_get_structure(caps, 0);
        gint width, height;
        gst_structure_get_int(structure, "width", &width);
        gst_structure_get_int(structure, "height", &height);
        GstBuffer *buffer = gst_sample_get_buffer(sample);
        GstMapInfo map;
        gst_buffer_map(buffer, &map, GST_MAP_READ);

        cv::Mat current_frame = cv::Mat(height, width, CV_8UC3, map.data);
        //cv::cvtColor(cv::Mat(height + height / 2, width, CV_8UC1, map.data), current_frame, cv::COLOR_YUV2BGR_NV12);

        std::cout << "frame1 rows: " << current_frame.rows << ", cols: " << current_frame.cols << std::endl;
        std::cout << "map.data: " << static_cast<void*>(map.data) << std::endl;

            if (!current_frame.empty()) {
                // Klonlama işlemi yerine doğrudan atama                
                frameMutex.lock();
                current_frame.copyTo(next_frame);
                frameMutex.unlock();
                std::cout << "frame2 rows: " << next_frame.rows << ", cols: " << next_frame.cols << std::endl;
                if (next_frame.data == nullptr) {
                    std::cerr << "Error: next_frame.data is nullptr after cloning." << std::endl;
                }
                g_print("Frame Received - Total Frames: %d\n", frame_counter);
                //cv::imwrite("Opencv.jpg", current_frame);
                //cv::imshow("opencv",current_frame);
                //current_frame.release();
            }
        
        
        gst_buffer_unmap(buffer, &map);
        gst_sample_unref(sample);
    }

    return GST_FLOW_OK;
}

void displayFrame() {
    cv::Mat readyFrame=cv::Mat::zeros(1280, 720, CV_8UC3);
    //cv::namedWindow("opencv", cv::WINDOW_AUTOSIZE);
    while (true) {
        if (!next_frame.empty()) {
            g_print("Here: \n");
            frameMutex.lock();
            next_frame.copyTo(readyFrame);
            frameMutex.unlock();
            g_print("Here-1: \n");
            //std::string fileName = "resimler/frame_" + std::to_string(frameCounter) + ".jpg";
            //cv::imwrite(fileName, readyFrame);
            frameCounter++;
            cv::imshow("opencv", readyFrame);
            char c=(char)cv::waitKey(25);
            if(c==27)
            {
                g_print("İnside-1: \n");
                break;
            }

            g_print("Here-2: \n");
        }
    }
}


int main(int argc, char *argv[]) {
    // Initialize GStreamer
    XInitThreads();
    g_main_context_push_thread_default(nullptr);
    gst_init(&argc, &argv);

    

    // Create elements
    GstElement *pipeline = gst_pipeline_new("mypipeline");
    GstElement *source = gst_element_factory_make("nvarguscamerasrc", "src");
    GstElement *capsfilter1 = gst_element_factory_make("capsfilter", "filter1");
    GstElement *nvvidconv1 = gst_element_factory_make("nvvidconv", "nvvidconv1");
    //GstElement *queue = gst_element_factory_make("queue", "queue");
    GstElement *videoconvert = gst_element_factory_make("videoconvert", "videoconvert");
    GstElement *appsink = gst_element_factory_make("appsink", "appsink");

    // Check for element creation errors
    if (!pipeline || !source || !capsfilter1 || !nvvidconv1 ||  !videoconvert || !appsink) {
        g_printerr("Not all elements could be created. Exiting.\n");
        return -1;
    }

    // Set element properties
    g_object_set(G_OBJECT(capsfilter1),
                 "caps",
                 gst_caps_from_string("video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=30/1"),
                 nullptr);

    g_object_set(G_OBJECT(videoconvert), "caps", gst_caps_from_string("video/x-raw, format=BGR"), nullptr);
    //g_object_set(G_OBJECT(queue), "max-size-time", 2000000000, "leaky", 2, "max-size-bytes",62914560 , "max-rate",1,  nullptr);


    // Configure appsink for efficient data retrieval
    g_object_set(appsink, "emit-signals", TRUE, "sync", false, "max-buffers", 30, "drop",false, nullptr); //"samplesperbuffer",1024, "num-buffers",-1,"blocksize",-1, "wait-on-eos",false,
    g_signal_connect(appsink, "new-sample", G_CALLBACK(on_new_sample), nullptr);

    // Add elements to pipeline and link them
    gst_bin_add_many(GST_BIN(pipeline), source, capsfilter1, nvvidconv1, videoconvert, appsink, nullptr);
    gst_element_link_many(source, capsfilter1, nvvidconv1, videoconvert, appsink, nullptr);

    // Add bus watch to listen for events
    GstBus *bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
    gst_bus_add_watch(bus, (GstBusFunc)bus_call, loop);
    gst_object_unref(bus);

    // Set pipeline to PLAYING state
    GstStateChangeReturn ret = gst_element_set_state(pipeline, GST_STATE_PLAYING);
    if (ret == GST_STATE_CHANGE_FAILURE) {
        g_printerr("Unable to set the pipeline to the playing state. Exiting.\n");
        gst_object_unref(pipeline);
        return -1;
    }
    std::thread displayThread(displayFrame);
    // Create GMainLoop for handling GStreamer events
    loop = g_main_loop_new(nullptr, FALSE);
    g_main_loop_run(loop);

    
    
    displayThread.join();

    // Clean up
    gst_element_set_state(pipeline, GST_STATE_NULL);
    gst_object_unref(pipeline);
    g_main_loop_unref(loop);

    return 0;
}

gstDebug.txt (8.4 MB)

rajupadhyay59 · January 7, 2024, 11:13pm

Hi,
This statement is able to print the output?

std::cout << "frame1 rows: " << current_frame.rows << ", cols: " << current_frame.cols << std::endl;
std::cout << "map.data: " << static_cast<void*>(map.data) << std::endl;

Can you for now try to not use your display thread. Instead try to save the images using cv::imwrite in the on_new_sample() function?

ziyaelri · January 8, 2024, 1:57pm

Hi,
Yes, this statement generates an output, and it increments the frame count every time the signal is triggered. I attempted to save frames using cv::imwrite within the on_new_sample() function as you recommended, but there was no improvement; each saved image displayed only a black screen.

I configured the waitKey() function to be triggered when the ‘q’ key is pressed, but it doesn’t work in any way.

I believe the issue is that OpenCV updates the screen without waitKey() working, but even when I inserted a delay, there was no change.

Therefore, I considered changing my approach by splitting the GStreamer pipeline into two using ‘tee’ (essentially creating two buffers). In the first ‘tee’, I intended to process the frames with OpenCV and display them with the second ‘tee’. However, I couldn’t redirect the frames processed by the first ‘tee’ to the second ‘tee’. Can you provide advice on this aspect based on the code I shared in my previous message?

Thank you.

rajupadhyay59 · January 8, 2024, 10:59pm

Why do you need to use cv::waitKey when you are saving images?
I too actually just used cv::imwrite when I was working with appsink to check if my images were being processed.

You’ll have to debug on your own, sorry.

ziyaelri · January 9, 2024, 12:01pm

I did not use cv::waitKey when I saved images. I wanted to check my process. For this reason, I used every method that I could use. But I could not too. Thank you for everything. If I solve this problem, I will share.

system · February 29, 2024, 1:59am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.