Deepstream6.0 deadlock

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) A10
• DeepStream Version 6.0
• JetPack Version (valid for Jetson only)
• TensorRT Version 8001
• NVIDIA GPU Driver Version (valid for GPU only) 470
• Issue Type( questions, new requirements, bugs) BUG
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Dynamically deleting a video stream has a chance to trigger a deadlock.
Trigger function:
state_return = gst_element_set_state (elem, GST_STATE_NULL);

context:


void
stop_release_source (NvDsSrcParentBin *bin, gint source_id)
{
    logger->info("stop source {}", source_id);
    GstStateChangeReturn state_return;
    GstElement *pipeline = bin->bin;
    gchar pad_name[16];
    GstPad *sinkpad = NULL;
    GstElement *elem = bin->sub_bins[source_id].bin;
    state_return =
        gst_element_set_state (elem, GST_STATE_NULL);
    switch (state_return) {
        case GST_STATE_CHANGE_SUCCESS:
            logger->info("STATE CHANGE SUCCESS");
            g_snprintf (pad_name, 15, "sink_%u", source_id);
            sinkpad = gst_element_get_static_pad (bin->streammux, pad_name);
            gst_pad_send_event (sinkpad, gst_event_new_flush_stop (FALSE));
            gst_element_release_request_pad (bin->streammux, sinkpad);
            logger->info("release SUCCESS");
            gst_object_unref (sinkpad);
            gst_bin_remove (GST_BIN (pipeline), elem);
//            source_id--;
//            bin->num_bins--;
            bin->sub_bins[source_id].have_eos = TRUE;
            break;
        case GST_STATE_CHANGE_FAILURE:
            logger->error("STATE CHANGE FAILURE");
            break;
        case GST_STATE_CHANGE_ASYNC:
            logger->info("STATE CHANGE ASYNC");
            do{
                state_return = gst_element_get_state (elem, NULL, NULL, GST_CLOCK_TIME_NONE);
                sleep(1);
            }while(state_return != GST_STATE_CHANGE_SUCCESS);
            g_snprintf (pad_name, 15, "sink_%u", source_id);
            sinkpad = gst_element_get_static_pad (bin->streammux, pad_name);
            gst_pad_send_event (sinkpad, gst_event_new_flush_stop (FALSE));
            gst_element_release_request_pad (bin->streammux, sinkpad);
            logger->info("release SUCCESS");
            gst_object_unref (sinkpad);
            gst_bin_remove (GST_BIN (pipeline), elem);
//            source_id--;
//            bin->num_bins--;
            bin->sub_bins[source_id].have_eos = TRUE;
            break;
        case GST_STATE_CHANGE_NO_PREROLL:
            logger->info("STATE CHANGE NO PREROLL");
            break;
        default:
            break;
    }
}

referred to:

Can the deadlock be reproduced by deepstream_reference_apps/runtime_source_add_delete at master · NVIDIA-AI-IOT/deepstream_reference_apps (github.com)? If so, how to reproduce? Please provide steps.

I didn’t try, I just copied the function to deepstream-test5.
I think the focus is how to use gst_element_set_state safety.

Please try it first. To compare the difference, you may know what is the root cause.

I tried, No deadlocks were triggered, I think the reason is that it doesn’t have multithreading.
Deepstream-test5 has many threads, so a deadlock is triggered .
But which thread conflicts with the delete stream, we need your help to check.

What do you need to be checked by us? Is there any issue you found related to deepstream modules? The problem is related to what you modified, you need to clarify why you change the code in this way.

Deepstream-app lacks efficient handling of offline cameras.
Runtime_source_add_delete is to solve the problem of abnormal program caused by camera offline.

Have you tried runtime_source_add_delete in Deepstream-test5?
Runtime_source_add_delete works in deepStream_reference_apps, but does not work in deepstream-test5,
I don’t think the deepStream-app module is compatible with the runtime_source_add_delete method for deleting streams .
You should check deepstream-app to see if runtime_source_add_delete is properly supported .

They are two different apps. You need to understand the code of runtime_source_add_delete and you may know how to implement the similiar function with deepstream-test5.

runtime_source_add_delete has shown that the interfaces are correct and working.

Did you see the code I uploaded? I think the method I used is fine, but it does occasionally deadlock on Deepstream-Test5. I’m pretty sure it was the multithreading of Deepstream-App that caused the deadlock.

So you can try to put the code in correct thread and give it proper thread synchronization protection.

My current confusion is that I do not know which thread triggered the deadlock. So I’m here to ask questions .If you can tell me what locks gst_element_set_state(elem, GST_STATE_NULL) triggers, I can fix the problem, too

It is not reasonable for us to debug every user’s own code and change. It is important to be familiar with gstreamer basic knowledge and coding skills before you start with deepstream.

This is not a good answer, deepStream was created to reduce the amount of work algorithmic engineers do on pipelines, and you can’t expect algorithmic engineers to be audio and video engineers. I’ve looked at gstreamer and it doesn’t help at all.

I’ve provided some stack information that I hope will enlighten you.

#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007fdf37ad90f4 in __GI___pthread_mutex_lock (mutex=0x7fdb340013d0) at ../nptl/pthread_mutex_lock.c:115
#2  0x00007fdf3940e87f in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#3  0x00007fdf3940f335 in gst_pad_set_active () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#4  0x00007fdf393ecf0d in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#5  0x00007fdf393ff884 in gst_iterator_fold () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#6  0x00007fdf393eda16 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#7  0x00007fdf393ef95e in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#8  0x00007fdf393efc06 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#9  0x00007fdf39713365 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#10 0x00007fddfade1f41 in gst_v4l2_video_dec_change_state () from target:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
#11 0x00007fdf393f1d5e in gst_element_change_state () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#12 0x00007fdf393f2499 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#13 0x00007fdf393cfa02 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#14 0x00007fdf23e50029 in ?? () from target:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#15 0x00007fdf393f1d5e in gst_element_change_state () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#16 0x00007fdf393f2499 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#17 0x00007fdf393cfa02 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#18 0x00007fdf23e6369a in ?? () from target:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#19 0x00007fdf393f1d5e in gst_element_change_state () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#20 0x00007fdf393f2499 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#21 0x00007fdf393cfa02 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#22 0x00007fdf393f1d5e in gst_element_change_state () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#23 0x00007fdf393f2045 in gst_element_change_state () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#24 0x00007fdf393f2499 in ?? () from target:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#25 0x0000559350e1e5b2 in stop_release_source(NvDsSrcParentBin*, int) ()
#26 0x0000559350e1f295 in event_thread_func_2(void*) ()
#27 0x00007fdf38e75e23 in ?? () from target:/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#28 0x00007fdf38e753a5 in g_main_context_dispatch () from target:/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#29 0x00007fdf38e75770 in ?? () from target:/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#30 0x00007fdf38e75a82 in g_main_loop_run () from target:/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#31 0x0000559350e21e92 in main ()

I got a reply from Gstreamer.They say it breaks while deactivating NVidia specific decoder in libgstnvvideo4linux2.so.