GStreamer pipeline recreation leads to segfault if the pipeline contains nvinfer node

Any attempt to create a new pipeline after the previous one was destroyed leads to segfault. Our application crashes with the following stack:

1   __GI_raise raise.c  54  0x7ffff2f81428 
2   __GI_abort abort.c  89  0x7ffff2f8302a 
3   __assert_fail_base assert.c 92  0x7ffff2f79bd7 
4   __GI___assert_fail assert.c 101 0x7ffff2f79c82 
5  nvinfer1::cudnn::WinogradConvActLayer::allocateResources(nvinfer1::cudnn::CommonContext const&) 0x7fff1a7f99d4 
6   nvinfer1::cudnn::Engine::initialize() 0x7fff1a7b2f77 
7   nvinfer1::cudnn::Engine::deserialize(void const *, unsigned long, nvinfer1::IGpuAllocator&, nvinfer1::IPluginFactory *) 0x7fff1a7b5cb5 
8   nvinfer1::Runtime::deserializeCudaEngine(void const *, unsigned long, nvinfer1::IPluginFactory *) 0x7fff1a7a5cd3 
9   NvllInfer_Init 0x7fff265e740f 
10  gst_nv_infer_start 0x7fff26c2f124 
...

The error is:

Assertion `convolutions.back().get()' failed.

The issue reproduces if apply this patch:

diff --git a/apps/deepstream-test1/deepstream_test1_app.c b/apps/deepstream-test1/deepstream_test1_app.c
index 434b9ee..28accb5 100644
--- a/apps/deepstream-test1/deepstream_test1_app.c
+++ b/apps/deepstream-test1/deepstream_test1_app.c
@@ -174,6 +174,9 @@ main (int argc, char *argv[])
 
   /* Standard GStreamer initialization */
   gst_init (&argc, &argv);
+
+  for (guint i = 0; i < 2; i++) {
+
   loop = g_main_loop_new (NULL, FALSE);
 
   /* Create gstreamer elements */
@@ -272,6 +275,9 @@ main (int argc, char *argv[])
   gst_object_unref (GST_OBJECT (pipeline));
   g_source_remove (bus_watch_id);
   g_main_loop_unref (loop);
+
+  }
+
   return 0;

to deepstream-test1 application provided with DeepStream.

The patch is suggested here https://devtalk.nvidia.com/default/topic/1037582/deepstream-for-tesla/restarting-pipeline-on-deepstream2-0/post/5271945/#5271945 and just initiializes and destroys the same pipeline in the cycle. On the second iteration we get segfault and the error:

deepstream-test1-app: nvll_infer.cpp:1190: bool GIE_Context::caffeToGIEModel(): Assertion `engine' failed.

The video used in both cases above is a sample provided with DS - sample_720p.*
Multiple test apps launched at the same time works fine, the issue is only with recreating the pipeline within a single process.

Any suggestions how to avoid those crashes?

@dmishin

Are you trying to run the pipeline with multiple file sources ? ( similar to the thread you have referred to in your question )

If yes, please try out the solution below. It basically runs inference on two streams in an infinite loop. Make sure to update the paths of two streams in the source_switch_thread function.

/*
 * Copyright (c) 2018 NVIDIA Corporation.  All rights reserved.
 *
 * NVIDIA Corporation and its licensors retain all intellectual property
 * and proprietary rights in and to this software, related documentation
 * and any modifications thereto.  Any use, reproduction, disclosure or
 * distribution of this software and related documentation without an express
 * license agreement from NVIDIA Corporation is strictly prohibited.
 *
 */

#include <glib.h>
#include <gst/gst.h>
#include <stdbool.h>

#include "gstnvdsmeta.h"

#define MAX_DISPLAY_LEN 64

#define PGIE_CLASS_ID_VEHICLE 0
#define PGIE_CLASS_ID_PERSON 2

gint frame_number = 0;
gboolean flag = true;
GMainLoop* loop = NULL;
gchar pgie_classes_str[4][32] = {"Vehicle", "TwoWheeler", "Person", "Roadsign"};

static gboolean source_switch_thread(gpointer* data)
{
    GstElement* pipeline = (GstElement*) data;
    GstElement* source = gst_bin_get_by_name(GST_BIN(pipeline), "file-source");
    GstElement* h264parser = gst_bin_get_by_name(GST_BIN(pipeline), "h264-parser");
    GstElement* sink = gst_bin_get_by_name(GST_BIN(pipeline), "nvvideo-renderer");
    gst_element_set_state(pipeline, GST_STATE_PAUSED);
    GstStateChangeReturn ret = GST_STATE_CHANGE_FAILURE;
    ret = gst_element_set_state(source, GST_STATE_NULL);
    if (ret == GST_STATE_CHANGE_FAILURE)
    {
        g_print("Unable to set state change for source element \n");
        g_main_loop_quit(loop);
    }
    if (flag)
    {
        g_object_set(G_OBJECT(source), "location", "Path/to/new/stream.h264",
                     NULL); /* Add path to new stream here */
        flag = false;
    }
    else
    {
        g_object_set(G_OBJECT(source), "location", "Path/to/new/stream-2.h264",
                     NULL); /* Add path to new stream here */
        flag = true;
    }
    gst_pad_activate_mode(gst_element_get_static_pad(h264parser, "sink"), GST_PAD_MODE_PUSH, true);
    gst_element_sync_state_with_parent(h264parser);
    gst_element_sync_state_with_parent(source);
    gst_element_set_state(sink, GST_STATE_READY);
    gst_element_sync_state_with_parent(sink);
    frame_number = 0;
    gst_element_set_state(pipeline, GST_STATE_PLAYING);
    return false;
}

static GstPadProbeReturn eos_probe_cb(GstPad* pad, GstPadProbeInfo* info, gpointer u_data)
{
    if (GST_EVENT_TYPE(GST_PAD_PROBE_INFO_DATA(info)) != GST_EVENT_EOS) { return GST_PAD_PROBE_OK; }

    gst_element_seek((GstElement*) u_data, 1.0, GST_FORMAT_TIME,
                     (GstSeekFlags)(GST_SEEK_FLAG_FLUSH), GST_SEEK_TYPE_SET, 0, GST_SEEK_TYPE_NONE,
                     GST_CLOCK_TIME_NONE);

    g_idle_add((GSourceFunc) source_switch_thread, u_data);
    return GST_PAD_PROBE_DROP;
}

/* osd_sink_pad_buffer_probe  will extract metadata received on OSD sink pad
 * and update params for drawing rectangle, object information etc. */

static GstPadProbeReturn osd_sink_pad_buffer_probe(GstPad* pad, GstPadProbeInfo* info,
                                                   gpointer u_data)
{

    GstMeta* gst_meta = NULL;
    NvDsMeta* nvdsmeta = NULL;
    gpointer state = NULL;
    static GQuark _nvdsmeta_quark = 0;
    GstBuffer* buf = (GstBuffer*) info->data;
    NvDsFrameMeta* frame_meta = NULL;
    guint num_rects = 0, rect_index = 0, l_index = 0;
    NvDsObjectParams* obj_meta = NULL;
    guint i = 0;
    NvOSD_TextParams* txt_params = NULL;
    guint vehicle_count = 0;
    guint person_count = 0;

    if (!_nvdsmeta_quark) _nvdsmeta_quark = g_quark_from_static_string(NVDS_META_STRING);

    while ((gst_meta = gst_buffer_iterate_meta(buf, &state)))
    {
        if (gst_meta_api_type_has_tag(gst_meta->info->api, _nvdsmeta_quark))
        {

            nvdsmeta = (NvDsMeta*) gst_meta;

            /* We are interested only in intercepting Meta of type
             * "NVDS_META_FRAME_INFO" as they are from our infer elements. */
            if (nvdsmeta->meta_type == NVDS_META_FRAME_INFO)
            {
                frame_meta = (NvDsFrameMeta*) nvdsmeta->meta_data;
                if (frame_meta == NULL)
                {
                    g_print("NvDS Meta contained NULL meta \n");
                    return GST_PAD_PROBE_OK;
                }

                /* We reset the num_strings here as we plan to iterate through the
                 *  the detected objects and form our own strings.
                 *  The pipeline generated strings shall be discarded.
                 */
                frame_meta->num_strings = 0;

                num_rects = frame_meta->num_rects;

                /* This means we have num_rects in frame_meta->obj_params,
                 * now lets iterate through them */

                for (rect_index = 0; rect_index < num_rects; rect_index++)
                {
                    /* Now using above information we need to form a text that should
                     * be displayed on top of the bounding box, so lets form it here. */

                    obj_meta = (NvDsObjectParams*) &frame_meta->obj_params[rect_index];

                    txt_params = &(obj_meta->text_params);
                    if (txt_params->display_text) g_free(txt_params->display_text);

                    txt_params->display_text = g_malloc0(MAX_DISPLAY_LEN);

                    g_snprintf(txt_params->display_text, MAX_DISPLAY_LEN, "%s ",
                               pgie_classes_str[obj_meta->class_id]);

                    if (obj_meta->class_id == PGIE_CLASS_ID_VEHICLE) vehicle_count++;
                    if (obj_meta->class_id == PGIE_CLASS_ID_PERSON) person_count++;

                    /* Now set the offsets where the string should appear */
                    txt_params->x_offset = obj_meta->rect_params.left;
                    txt_params->y_offset = obj_meta->rect_params.top - 25;

                    /* Font , font-color and font-size */
                    txt_params->font_params.font_name = "Arial";
                    txt_params->font_params.font_size = 10;
                    txt_params->font_params.font_color.red = 1.0;
                    txt_params->font_params.font_color.green = 1.0;
                    txt_params->font_params.font_color.blue = 1.0;
                    txt_params->font_params.font_color.alpha = 1.0;

                    /* Text background color */
                    txt_params->set_bg_clr = 1;
                    txt_params->text_bg_clr.red = 0.0;
                    txt_params->text_bg_clr.green = 0.0;
                    txt_params->text_bg_clr.blue = 0.0;
                    txt_params->text_bg_clr.alpha = 1.0;

                    frame_meta->num_strings++;
                }
            }
        }
    }
    g_print(
        "Frame Number = %d Number of objects = %d "
        "Vehicle Count = %d Person Count = %d\n",
        frame_number, num_rects, vehicle_count, person_count);
    frame_number++;

    return GST_PAD_PROBE_OK;
}

static gboolean bus_call(GstBus* bus, GstMessage* msg, gpointer data)
{
    GMainLoop* loop = (GMainLoop*) data;
    switch (GST_MESSAGE_TYPE(msg))
    {
    case GST_MESSAGE_EOS:
        g_print("End of stream\n");
        g_main_loop_quit(loop);
        break;
    case GST_MESSAGE_ERROR:
    {
        gchar* debug;
        GError* error;
        gst_message_parse_error(msg, &error, &debug);
        g_printerr("ERROR from element %s: %s\n", GST_OBJECT_NAME(msg->src), error->message);
        g_free(debug);
        g_printerr("Error: %s\n", error->message);
        g_error_free(error);
        g_main_loop_quit(loop);
        break;
    }
    default: break;
    }
    return TRUE;
}

int main(int argc, char* argv[])
{

    GstElement *pipeline = NULL, *source = NULL, *h264parser = NULL, *decoder = NULL, *sink = NULL,
               *pgie = NULL, *nvvidconv = NULL, *nvosd = NULL, *filter1 = NULL, *filter2 = NULL;
    GstBus* bus = NULL;
    guint bus_watch_id;
    GstCaps *caps1 = NULL, *caps2 = NULL;
    gulong osd_sink_probe_id = 0, osd_src_probe_id = 0;
    GstPad *osd_sink_pad = NULL, *osd_src_pad = NULL;

    /* Check input arguments */
    if (argc != 2)
    {
        g_printerr("Usage: %s <H264 filename>\n", argv[0]);
        return -1;
    }

    /* Standard GStreamer initialization */
    gst_init(&argc, &argv);
    loop = g_main_loop_new(NULL, FALSE);

    /* Create gstreamer elements */
    /* Create Pipeline element that will form a connection of other elements */
    pipeline = gst_pipeline_new("dstest1-pipeline");

    /* Source element for reading from the file */
    source = gst_element_factory_make("filesrc", "file-source");

    /* Since the data format in the input file is elementary h264 stream,
     * we need a h264parser */
    h264parser = gst_element_factory_make("h264parse", "h264-parser");

    /* Use nvdec_h264 for hardware accelerated decode on GPU */
    decoder = gst_element_factory_make("nvdec_h264", "nvh264-decoder");

    /* Use nvinfer to run inferencing on decoder's output,
     * behaviour of inferencing is set through config file */
    pgie = gst_element_factory_make("nvinfer", "primary-nvinference-engine");

    /* Use convertor to convert from NV12 to RGBA as required by nvosd */
    nvvidconv = gst_element_factory_make("nvvidconv", "nvvideo-converter");

    /* Create OSD to draw on the converted RGBA buffer */
    nvosd = gst_element_factory_make("nvosd", "nv-onscreendisplay");

    /* Finally render the osd output */
    sink = gst_element_factory_make("nveglglessink", "nvvideo-renderer");

    /* caps filter for nvvidconv to convert NV12 to RGBA as nvosd expects input
     * in RGBA format */
    filter1 = gst_element_factory_make("capsfilter", "filter1");
    filter2 = gst_element_factory_make("capsfilter", "filter2");
    if (!pipeline || !source || !h264parser || !decoder || !pgie || !filter1 || !nvvidconv
        || !filter2 || !nvosd || !sink)
    {
        g_printerr("One element could not be created. Exiting.\n");
        return -1;
    }

    /* we set the input filename to the source element */
    g_object_set(G_OBJECT(source), "location", argv[1], NULL);

    /* Set all the necessary properties of the nvinfer element,
     * the necessary ones are : */
    g_object_set(G_OBJECT(pgie), "config-file-path", "dstest1_pgie_config.txt", NULL);

    /* we set the osd properties here */
    g_object_set(G_OBJECT(nvosd), "font-size", 15, NULL);

    /* we add a message handler */
    bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
    bus_watch_id = gst_bus_add_watch(bus, bus_call, loop);
    gst_object_unref(bus);

    /* Set up the pipeline */
    /* we add all elements into the pipeline */
    gst_bin_add_many(GST_BIN(pipeline), source, h264parser, decoder, pgie, filter1, nvvidconv,
                     filter2, nvosd, sink, NULL);
    caps1 = gst_caps_from_string("video/x-raw(memory:NVMM), format=NV12");
    g_object_set(G_OBJECT(filter1), "caps", caps1, NULL);
    gst_caps_unref(caps1);
    caps2 = gst_caps_from_string("video/x-raw(memory:NVMM), format=RGBA");
    g_object_set(G_OBJECT(filter2), "caps", caps2, NULL);
    gst_caps_unref(caps2);

    /* we link the elements together */
    /* file-source -> h264-parser -> nvh264-decoder ->
     * nvinfer -> filter1 -> nvvidconv -> filter2 -> nvosd -> video-renderer */
    gst_element_link_many(source, h264parser, decoder, pgie, filter1, nvvidconv, filter2, nvosd,
                          sink, NULL);

    /* Lets add probe to get informed of the meta data generated, we add probe to
     * the sink pad of the osd element, since by that time, the buffer would have
     * had got all the metadata. */
    osd_sink_pad = gst_element_get_static_pad(nvosd, "sink");
    if (!osd_sink_pad)
        g_print("Unable to get OSD sink pad\n");
    else
        osd_sink_probe_id = gst_pad_add_probe(osd_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
                                              osd_sink_pad_buffer_probe, NULL, NULL);

    osd_src_pad = gst_element_get_static_pad(nvosd, "src");
    if (!osd_src_pad)
        g_print("Unable to get OSD src pad \n");
    else
        osd_src_probe_id = gst_pad_add_probe(osd_src_pad, GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM,
                                             eos_probe_cb, pipeline, NULL);
    /* Set the pipeline to "playing" state */
    g_print("Now playing: %s\n", argv[1]);
    gst_element_set_state(pipeline, GST_STATE_PLAYING);

    /* Wait till pipeline encounters an error or EOS */
    g_print("Running...\n");
    g_main_loop_run(loop);

    /* Out of the main loop, clean up nicely */
    g_print("Returned, stopping playback\n");
    gst_element_set_state(pipeline, GST_STATE_NULL);
    g_print("Deleting pipeline\n");
    gst_object_unref(GST_OBJECT(pipeline));
    g_source_remove(bus_watch_id);
    g_main_loop_unref(loop);
    return 0;
}

Let us know if your objective was something different.

@NvCJR

Thank you, this example is really useful and we certainly will use it. I’ll try to adapt it for AppSrc which we use in our application (btw maybe you have an example for AppSrc with dynamic stream parameter change handling (codec/resolution)?).

But it doesn’t answer for original question - why does application crash when pipeline is destroyed and then created again? It’s quite unexpected behavior and looks like a bug. For some reason we need to destroy and create pipelines again sometimes.

The patch is suggested here https://devtalk.nvidia.com/default/topic/1037582/deepstream-for-tesla/restarting-pipeline-on-deepstream2-0/post/5271945/#5271945 and just initiializes and destroys the same pipeline in the cycle. On the second iteration we get segfault and the error:
deepstream-test1-app: nvll_infer.cpp:1190: bool GIE_Context::caffeToGIEModel(): Assertion `engine’ failed.

you need sleep for 1 seconds or more between each cycle

Hi dmishin,

did you get the solution for this problem?
I am also doing the same and getting the segmentation fault on second time.
Please suggest any solution.

Thanks & Regards,
Shubham

Where you run on? Tegra or x86 platform? i can not get the segmentation error after second time reinit pipeline on Tegra xavier, you use customized test1 app or just apply patch from https://devtalk.nvidia.com/default/topic/1037582/deepstream-for-tesla/restarting-pipeline-on-deepstream2-0/post/5271945/#5271945 based on test1 app?

I am running it on Tegra platform.
There are 2 cases-

  1. When I run test3 application with default resnet model it works fine for every time.
  2. When I run test3 application with ssd inception model it get crashed on 2nd time with seg fault.

I also debug this issue and I think the problem is with objectDetectionSSD lib.

test1 app is for default resnet model, if you want to use objectDetectionSSD, i think you need to custom the test1 app code accordingly.

Hello, is it possible for modifying the test1 app for using my own trained model ?
The pipeline is the same as test1 app but I’d like to apply inference process with my model.
Any advice?
Thanks.

refer to this documentation:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/#page/DeepStream%2520Development%2520Guide%2Fdeepstream_custom_model.html%23wwpID0ERHA