Gst_element_set_state() is hanging while setting the element to NULL

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hardware Platform (Jetson / GPU): Tesla T4
Deepstream Version: 6.1
NVIDIA GPU Driver Version: " NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6"
Issue Type: questions

We are currently developing a multi-primary inference application using deepstream.

Our top level architecture of application look likes this.

Pipeline:

> 			      |-> [jpeg-encode-bin]
> [source-bin]-->|	
> 			      | ->[inference-bin]

Source-bin:

[rtspsrc -> rtph264depay -> h264parse ->nvv4l2decoder -> queue -> tee] -> | (ghost_pad_jpegenc)
															            -> | (ghost_pad_inference)

Jpeg-bin:

(ghost_pad_jpegenc) | [nvvideoconvert -> nvjpegenc(custom-plugin) -> nvmsgbroker]


Inference-bin:

(ghost_pad_inference) | [nvstreammux -> nvinfer -> nvtracker -> nvmsgbroker ]


Issue:

  • We are having various rtsp source (cameras). In some of the cameras source-bin is not able to decode the that particular stream.

  • When we encounter this issue and tries to de-init the source-bin using gst_element_set_state(sourcebin, GST_STATE_NULL)

  • My thread hangs here and does not come out of it. If we replace the nvv4l2decoder with avdec_h264(s/w decoder) we are not facing this pipeline hang.

  • I have also tried sending various events to sourcebin such as :
    gst_element_send_event(sourceElements->sourceBin, gst_event_new_eos());
    gst_element_send_event(sourceElements->sourceBin, gst_nvevent_new_stream_reset(0));
    gst_element_send_event(sourceElements->sourceBin, gst_event_new_flush_start());

    But nothing helps here and application is not able to de-init the pipeline and stucks.

I am stuck in this issue from more than a month now. Any help or direction would greatly appreciated.

Can you provide a simple sample to reproduce the hang?

Let me try but actually, we are adding and deleting sources using REST APIs from UI and also this issue is happing with certain camera streams which might be not reproducible at your end.

But if it is possible to prepare some sample code out of this I will provide that.

live555MediaServer (1.0 MB)
video2.264 (30.8 MB)
Hello @Fiona.Chen,

Sample code would be difficult but I am providing you a sample gst-launch pipeline that replicates similar behavior.

Setup: Deepstream 6.1 container


Software Decoder

gst-launch-1.0 rtspsrc location="rtsp://10.77.53.45:8554/video2.264" ! rtph264depay ! h264parse ! avdec_h264 ! nvvideoconvert ! jpegenc ! multifilesink location=image%d.jpeg

Observation: Above pipeline is working and dumping the jpeg images also on the CTRL+C interrupt gst-launch is terminating gracefully.


nvv4l2decoder Decoder

gst-launch-1.0 rtspsrc location="rtsp://10.77.53.45:8554/video2.264" ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvideoconvert ! jpegenc ! multifilesink location=image%d.jpeg

Observation: Above pipeline is not working and not getting jpeg images. Also on the CTRL+C interrupt gst-launch is not terminating gracefully and hangs on Setting pipeline to READY. Logs are as below:

gst-launch-1.0 rtspsrc location=“rtsp://10.77.53.45:8554/video2.264” async-handling=true udp-reconnect=false ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvideoconvert ! jpegenc ! multifilesink location=image%d.jpeg
Setting pipeline to PAUSED …
0:00:00.057443593 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057469861 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MJPG
0:00:00.057475881 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057480664 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MJPG
0:00:00.057495993 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057510590 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG4
0:00:00.057516648 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057525219 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG4
0:00:00.057537029 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057547977 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG2
0:00:00.057554611 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057559090 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG2
0:00:00.057570703 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057577474 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H265
0:00:00.057585047 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057591554 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H265
0:00:00.057601726 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057609540 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP90
0:00:00.057613315 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057622069 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP90
0:00:00.057632897 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057639464 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP80
0:00:00.057643578 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057650512 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP80
0:00:00.057659414 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057664303 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H264
0:00:00.057667976 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
0:00:00.057673133 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H264
0:00:00.058084461 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:00.058095808 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat NM12
0:00:00.058104598 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
0:00:00.058113468 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat NM12
0:00:00.058124776 4076254 0x55db18203c30 WARN v4l2 gstv4l2object.c:2388:gst_v4l2_object_add_interlace_mode:0x55db181316e0 Failed to determine interlace mode
Pipeline is live and does not need PREROLL …
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://10.77.53.45:8554/video2.264
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING …
New clock: GstSystemClock
Redistribute latency…
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
0:00:00.167613448 4076254 0x7f3de403c5e0 FIXME default gstutils.c:3981:gst_pad_create_stream_id_internal:fakesrc0:src Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Progress: (request) Sent PLAY request
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:02.447540467
Setting pipeline to PAUSED …
Setting pipeline to READY …

I am attaching RTSP simulator and video for you to reproduce the issue.

The issue here is we are not able to recover from this state in our application using any kind of events such as EOS, flushing, or anything.

When I tried iterating over child plugins in decodebin, and setting them to NULL, it hangs at nvv4l2decoder.
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:292 decodebin3-3 found
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:292 nvv4l2decoder3 found
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:303 setting nvv4l2decoder3 to null

I can reproduce this issue. We will look into it.

1 Like