Gst_element_set_state() is hanging while setting the element to NULL

hardik.panchal · July 21, 2022, 12:24pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hardware Platform (Jetson / GPU): Tesla T4
Deepstream Version: 6.1
NVIDIA GPU Driver Version: " NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6"
Issue Type: questions

We are currently developing a multi-primary inference application using deepstream.

Our top level architecture of application look likes this.

Pipeline:

> 			      |-> [jpeg-encode-bin]
> [source-bin]-->|	
> 			      | ->[inference-bin]

Source-bin:

[rtspsrc -> rtph264depay -> h264parse ->nvv4l2decoder -> queue -> tee] -> | (ghost_pad_jpegenc)
															            -> | (ghost_pad_inference)

Jpeg-bin:

(ghost_pad_jpegenc) | [nvvideoconvert -> nvjpegenc(custom-plugin) -> nvmsgbroker]

Inference-bin:

(ghost_pad_inference) | [nvstreammux -> nvinfer -> nvtracker -> nvmsgbroker ]

Issue:

We are having various rtsp source (cameras). In some of the cameras source-bin is not able to decode the that particular stream.
When we encounter this issue and tries to de-init the source-bin using gst_element_set_state(sourcebin, GST_STATE_NULL)
My thread hangs here and does not come out of it. If we replace the nvv4l2decoder with avdec_h264(s/w decoder) we are not facing this pipeline hang.
I have also tried sending various events to sourcebin such as :
gst_element_send_event(sourceElements->sourceBin, gst_event_new_eos());
gst_element_send_event(sourceElements->sourceBin, gst_nvevent_new_stream_reset(0));
gst_element_send_event(sourceElements->sourceBin, gst_event_new_flush_start());

But nothing helps here and application is not able to de-init the pipeline and stucks.

I am stuck in this issue from more than a month now. Any help or direction would greatly appreciated.

Fiona.Chen · July 21, 2022, 1:09pm

Can you provide a simple sample to reproduce the hang?

hardik.panchal · July 21, 2022, 1:50pm

Let me try but actually, we are adding and deleting sources using REST APIs from UI and also this issue is happing with certain camera streams which might be not reproducible at your end.

But if it is possible to prepare some sample code out of this I will provide that.

hardik.panchal · July 21, 2022, 3:14pm

live555MediaServer (1.0 MB)
video2.264 (30.8 MB)
Hello @Fiona.Chen,

Sample code would be difficult but I am providing you a sample gst-launch pipeline that replicates similar behavior.

Setup: Deepstream 6.1 container

Software Decoder

gst-launch-1.0 rtspsrc location="rtsp://10.77.53.45:8554/video2.264" ! rtph264depay ! h264parse ! avdec_h264 ! nvvideoconvert ! jpegenc ! multifilesink location=image%d.jpeg

Observation: Above pipeline is working and dumping the jpeg images also on the CTRL+C interrupt gst-launch is terminating gracefully.

nvv4l2decoder Decoder

gst-launch-1.0 rtspsrc location="rtsp://10.77.53.45:8554/video2.264" ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvideoconvert ! jpegenc ! multifilesink location=image%d.jpeg

Observation: Above pipeline is not working and not getting jpeg images. Also on the CTRL+C interrupt gst-launch is not terminating gracefully and hangs on Setting pipeline to READY. Logs are as below:

gst-launch-1.0 rtspsrc location=“rtsp://10. Setting pipeline to PAUSED …
0:00:00.057443593 4076254 0x55db18203c30 WARN 0:00:00.057469861 4076254 0x55db18203c30 WARN 0:00:00.057475881 4076254 0x55db18203c30 WARN 0:00:00.057480664 4076254 0x55db18203c30 WARN 0:00:00.057495993 4076254 0x55db18203c30 WARN 0:00:00.057510590 4076254 0x55db18203c30 WARN 0:00:00.057516648 4076254 0x55db18203c30 WARN 0:00:00.057525219 4076254 0x55db18203c30 WARN 0:00:00.057537029 4076254 0x55db18203c30 WARN 0:00:00.057547977 4076254 0x55db18203c30 WARN 0:00:00.057554611 4076254 0x55db18203c30 WARN 0:00:00.057559090 4076254 0x55db18203c30 WARN 0:00:00.057570703 4076254 0x55db18203c30 WARN 0:00:00.057577474 4076254 0x55db18203c30 WARN 0:00:00.057585047 4076254 0x55db18203c30 WARN 0:00:00.057591554 4076254 0x55db18203c30 WARN 0:00:00.057601726 4076254 0x55db18203c30 WARN 0:00:00.057609540 4076254 0x55db18203c30 WARN 0:00:00.057613315 4076254 0x55db18203c30 WARN 0:00:00.057622069 4076254 0x55db18203c30 WARN 0:00:00.057632897 4076254 0x55db18203c30 WARN 0:00:00.057639464 4076254 0x55db18203c30 WARN 0:00:00.057643578 4076254 0x55db18203c30 WARN 0:00:00.057650512 4076254 0x55db18203c30 WARN 0:00:00.057659414 4076254 0x55db18203c30 WARN 0:00:00.057664303 4076254 0x55db18203c30 WARN 0:00:00.057667976 4076254 0x55db18203c30 WARN 0:00:00.057673133 4076254 0x55db18203c30 WARN 0:00:00.058084461 4076254 0x55db18203c30 WARN 0:00:00.058095808 4076254 0x55db18203c30 WARN 0:00:00.058104598 4076254 0x55db18203c30 WARN 0:00:00.058113468 4076254 0x55db18203c30 WARN 0:00:00.058124776 4076254 0x55db18203c30 WARN Pipeline is live and does not need PREROLL Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://10.7 Progress: (open) Retrieving server options
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING …
New clock: GstSystemClock
Redistribute latency…
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
0:00:00.167613448 4076254 0x7f3de403c5e0 FIXME Progress: (request) Sent PLAY request
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:02.447540467
Setting pipeline to PAUSED …
Setting pipeline to READY … 77.53.45:8554/video2.264” async-handling=true udp-reconnect=false ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvideoconvert ! jpegenc ! multifilesink location=image%d.jpeg
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MJPG
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MJPG
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG4
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG4
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG2
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG2
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H265
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H265
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP90
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP90
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP80
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP80
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H264
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H264
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2937:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat NM12
v4l2 gstv4l2object.c:3051:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
v4l2 gstv4l2object.c:2943:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat NM12
v4l2 gstv4l2object.c:2388:gst_v4l2_object_add_interlace_mode:0x55db181316e0 Failed to determine interlace mode
…
7.53.45:8554/video2.264
> default gstutils.c:3981:gst_pad_create_stream_id_internal:fakesrc0:src Creating random stream-id, consider implementing a deterministic way of creating a stream-id

I am attaching RTSP simulator and video for you to reproduce the issue.

hardik.panchal · August 5, 2022, 6:07am

The issue here is we are not able to recover from this state in our application using any kind of events such as EOS, flushing, or anything.

When I tried iterating over child plugins in decodebin, and setting them to NULL, it hangs at nvv4l2decoder.
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:292 decodebin3-3 found
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:292 nvv4l2decoder3 found
2022-08-05 05:56:26,168 [INFO] DeepStreamVideoCapture.cpp:binIterator:303 setting nvv4l2decoder3 to null

Fiona.Chen · August 8, 2022, 3:32pm

I can reproduce this issue. We will look into it.

Fiona.Chen · August 10, 2022, 12:34pm

Live555 with raw H264 data can not guarantee IDR frame with headers. Current hardware decoder can not handle such case. Can you try with the following pipeline?

gst-launch-1.0 rtspsrc location= rtsp://10.24.219.254:8554/1.264 !  rtph264depay ! h264parse config-interval=-1 ! nvv4l2decoder  ! nveglglessink

hardik.panchal · August 10, 2022, 12:57pm

Hi @Fiona.Chen ,

Thanks for looking into the issue.

I have tried setting the ‘config-interval=-1’ property for h264parse but still the issue still persist at my end.

I am using filesink instead of ‘nveglglessink’ as we do not have display attached to the system, but I guess that would not make any difference here.

Fiona.Chen · August 10, 2022, 1:11pm

We have fix the problem internally. Please wait for the next release.

hardik.panchal · August 11, 2022, 8:53am

Thank you @Fiona.Chen for the update.

system · August 25, 2022, 8:53am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.