Releasing streammux pad hangs gstreamer pipeline when adding and deleting rtsp source multiple times

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.1.1
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.2.2
• NVIDIA GPU Driver Version (valid for GPU only) 515.65.01
• Issue Type( questions, new requirements, bugs) bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

Created pipeline based on deepstream_reference_apps/runtime_source_add_delete at master · NVIDIA-AI-IOT/deepstream_reference_apps · GitHub. Added around 10-11 rtsp sources and periodically doing add/remove cameras.

  • observed that after couple of add/remove cameras, DS pipeline got stuck in release pad for stream max
    • Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

One more scenario is : Add invalid RTSP source along with other RTSP source hangs the streammux.

Below is code for remove source.

def stop(self):
    """
    This method helps to stop camera source

    :return: None
    """
    if self.source_bin:
        # Retrieve sink pad to be released,
        # https: // forums.developer.nvidia.com / t / deepstream - pipeline - hangs - on - gst - element - set - state - uridecodebin / 169009
        padname = "%s_%u" % ("sink", self.stream_index)
        sinkpad = Gstreamer.get_static_pad(self.streammux, padname)
        if sinkpad:
            Gstreamer.send_new_eos_event(sinkpad)
            #time.sleep(Stream.SLEEP_TIME_FOR_FLUSH)

            state_return = Gstreamer.set_state(self.source_bin, Gstreamer.null_state)
            if state_return in [
                Gstreamer.change_return_success,
                Gstreamer.change_return_async
            ]:
                Gstreamer.send_flush_stop_event(sinkpad)
                Gstreamer.release_request_pad(self.streammux, sinkpad)

                # Remove the source bin from the pipeline
                Gstreamer.remove_element_from_pipeline(self.pipeline, self.source_bin)

                logger.info(f"source removed successfully : {self.stream_id}")
            elif state_return == Gstreamer.change_return_failure:
                logger.info(f"state change failure : {self.stream_id}")
3

Can you try the python sample deepstream_python_apps/apps/runtime_source_add_delete at master · NVIDIA-AI-IOT/deepstream_python_apps (github.com)?

4

Thanks Fiona.
yes, I will try and update it. I have created application based on this deepstream_python_apps/apps/runtime_source_add_delete at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub example only. trying on this application requires some customisation as my application keep running all sources for long run and I I can delete few camera and add few more using UI interface.

is there way I can enable Nvidia elements logs to see where exactly it stuck. if I enabled logs with GST_DEBUG with level > 3 then it prints lots of log which make difficult for debugging.

I found another hang with nvtracker element. I have added probe sink and source pad on tracker and doing additional and removal of camera at regular interval. I can see sink pad logs are coming from tracker but not source pad.

I have observed it hangs entire pipeline. is there way to check where it can hang in nvtracker element.

5

How many cameras do you have? Have you set the maximum source number as the nvstreammux batch-size and set nvmultistreammux columns and rows correctly?

The GST_DEBUG can set element level log either, so you must know which element you want to track. Basic tutorial 11: Debugging tools

The better way is to find out a simple app to reproduce the hang so that we can reproduce in our side.

6

Have added 8 cameras and doing addition and deletion of camera one by one , making sure 4 cameras remain active into pipeline.

yes, maximum source number into nvstreammux batch-size set to 32. not using nvmultistreammux. below is pipeline.

streammux → nvinferserver (PGIE) → queue → tracker → queue → nvinferserver (SGIE) → queue → nvmsgconv → nvmsgbroker

We are using triton-server over grpc.

7

The better way is to find out a simple app to reproduce the hang so that we can reproduce in our side.

8

Let me customise deepstream_python_apps/apps/runtime_source_add_delete at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub this and share with steps to reproduce this issue.

9

Hello Fiona,

I have updated deepstream-test3 app with runtime add/delete source and testing with nvinferserver configuration to reproduce above issue and application work fine until I delete one of source randomly and perf reports 0 fps for all sources. Looks like all source bin went to pause state.

Below are logs for same and also attached deepstream-test3.tar.gz (deepstream app with config). peoplenet model I could not upload as size is >100MB. I created peoplenet model engine file with batch size 32. I have used peoplenet model suggested in README file in deepstream-test3.

root@ai_service:/app#
root@ai_service:/app#
root@ai_service:/app#
root@ai_service:/app# cd /opt/nvidia/deepstream/deepstream/
LICENSE.txt README entrypoint.sh samples/ user_additional_install.sh
LicenseAgreement.pdf README.rhel install.sh sources/ version
LicenseAgreementContainer.pdf bin/ lib/ uninstall.sh
root@ai_service:/app# cd /opt/nvidia/deepstream/deepstream/sources/
SONYCAudioClassifier/ apps/ gst-plugins/ libs/ objectDetector_SSD/ tools/
TritonOnnxYolo/ deepstream_python_apps/ includes/ objectDetector_FasterRCNN/ objectDetector_Yolo/ tracker_DeepSORT/
root@ai_service:/app# cd /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/
.git/ .python-app-pipeline.png 3rdparty/ HOWTO.md README.md apps/ docs/ tests/
.gitmodules .test3-app.png FAQ.md LICENSE THIRD_PARTY_LICENSE bindings/ notebooks/
root@ai_service:/app# cd /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/
README deepstream-nvdsanalytics/ deepstream-ssd-parser/ deepstream-test3/
common/ deepstream-opticalflow/ deepstream-test1/ deepstream-test3.tar.gz
deepstream-demux-multi-in-multi-out/ deepstream-preprocess-test/ deepstream-test1-rtsp-out/ deepstream-test4/
deepstream-imagedata-multistream/ deepstream-rtsp-in-rtsp-out/ deepstream-test1-usbcam/ runtime_source_add_delete/
deepstream-imagedata-multistream-redaction/ deepstream-segmentation/ deepstream-test2/
root@ai_service:/app# cd /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/deepstream-test3
root@ai_service:/opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/deepstream-test3# ls
README config_infer_primary_peoplenet.txt config_triton_grpc_infer_primary_peoplenet.txt deepstream_test_3.py dstest_tracker_config.txt
config.pbtxt config_tracker_NvDCF_perf.yml config_triton_infer_primary_peoplenet.txt dstest3_pgie_config.txt tracker_config.yml
root@ai_service:/opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/deepstream-test3# python3 deepstream_test_3.py -i rtsp://20.166.69.85:8554/live/stream1 rtsp://20.166.69.85:8554/live/stream2 rtsp://20.166.69.85:8554/live/stream3 rtsp://20.166.69.85:8554/live/stream4 rtsp://20.166.69.85:8554/live/stream5 rtsp://20.166.69.85:8554/live/stream6 rtsp://20.166.69.85:8554/live/stream7 rtsp://20.166.69.85:8554/live/stream8 rtsp://20.166.69.85:8554/live/stream9 rtsp://20.166.69.85:8554/live/stream10 --pgie nvinferserver -c config_triton_infer_primary_peoplenet.txt --no-display
{‘input’: [‘rtsp://20.166.69.85:8554/live/stream1’, ‘rtsp://20.166.69.85:8554/live/stream2’, ‘rtsp://20.166.69.85:8554/live/stream3’, ‘rtsp://20.166.69.85:8554/live/stream4’, ‘rtsp://20.166.69.85:8554/live/stream5’, ‘rtsp://20.166.69.85:8554/live/stream6’, ‘rtsp://20.166.69.85:8554/live/stream7’, ‘rtsp://20.166.69.85:8554/live/stream8’, ‘rtsp://20.166.69.85:8554/live/stream9’, ‘rtsp://20.166.69.85:8554/live/stream10’], ‘configfile’: ‘config_triton_infer_primary_peoplenet.txt’, ‘pgie’: ‘nvinferserver’, ‘no_display’: True, ‘file_loop’: False, ‘disable_probe’: False, ‘silent’: False}
Creating Pipeline

Creating streamux

Creating source_bin 0

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream1]
source-bin-00
Creating source_bin 1

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream2]
source-bin-01
Creating source_bin 2

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream3]
source-bin-02
Creating source_bin 3

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream4]
source-bin-03
Creating source_bin 4

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream5]
source-bin-04
Creating source_bin 5

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream6]
source-bin-05
Creating source_bin 6

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream7]
source-bin-06
Creating source_bin 7

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream8]
source-bin-07
Creating source_bin 8

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream9]
source-bin-08
Creating source_bin 9

Creating uridecodebin for [rtsp://20.166.69.85:8554/live/stream10]
source-bin-09
Creating Pgie

Creating nvtracker

Creating tiler

Creating nvvidconv

Creating nvosd

Creating Fakesink

At least one of the sources is live
WARNING: Overriding infer-config batch-size 0 with number of sources 10

Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
0 : rtsp://20.166.69.85:8554/live/stream1
1 : rtsp://20.166.69.85:8554/live/stream2
2 : rtsp://20.166.69.85:8554/live/stream3
3 : rtsp://20.166.69.85:8554/live/stream4
4 : rtsp://20.166.69.85:8554/live/stream5
5 : rtsp://20.166.69.85:8554/live/stream6
6 : rtsp://20.166.69.85:8554/live/stream7
7 : rtsp://20.166.69.85:8554/live/stream8
8 : rtsp://20.166.69.85:8554/live/stream9
9 : rtsp://20.166.69.85:8554/live/stream10
Starting pipeline

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is OFF
[NvMultiObjectTracker] Initialized
0:00:01.932453910 3360 0x29b5d60 WARN nvinferserver gstnvinferserver_impl.cpp:315:validatePluginConfig: warning: Configuration file batch-size reset to: 10
WARNING: infer_proto_utils.cpp:144 auto-update preprocess.network_format to IMAGE_FORMAT_RGB
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: peoplenet
Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Warning: gst-library-error-quark: Configuration file batch-size reset to: 10 (5): gstnvinferserver_impl.cpp(315): validatePluginConfig (): /GstPipeline:pipeline0/GstNvInferServer:primary-inference
Decodebin child added: decodebin0

Decodebin child added: decodebin1

Decodebin child added: decodebin2

Decodebin child added: rtph264depay1

Decodebin child added: rtph264depay2
Decodebin child added: rtph264depay0

Decodebin child added: decodebin3

Decodebin child added: h264parse0

Decodebin child added: h264parse1

Decodebin child added: capsfilter0

Decodebin child added: rtph264depay3

Decodebin child added: h264parse2

Decodebin child added: capsfilter1

Decodebin child added: h264parse3

Decodebin child added: capsfilter2

Decodebin child added: capsfilter3

Decodebin child added: nvv4l2decoder0

Decodebin child added: nvv4l2decoder2

Decodebin child added: nvv4l2decoder1

Decodebin child added: nvv4l2decoder3

Decodebin child added: decodebin4

Decodebin child added: rtph264depay4

Decodebin child added: decodebin5

Decodebin child added: h264parse4
Decodebin child added: decodebin6

Decodebin child added: decodebin7

Decodebin child added: rtph264depay5

Decodebin child added: rtph264depay6

Decodebin child added: capsfilter4

Decodebin child added: decodebin8

Decodebin child added: decodebin9

Decodebin child added: h264parse5

Decodebin child added: h264parse6

Decodebin child added: nvv4l2decoder4

Decodebin child added: rtph264depay7

Decodebin child added: rtph264depay8

Decodebin child added: h264parse7

Decodebin child added: capsfilter5

Decodebin child added: nvv4l2decoder5

Decodebin child added: rtph264depay9

Decodebin child added: h264parse8

Decodebin child added: capsfilter6

Decodebin child added: nvv4l2decoder6
Decodebin child added: capsfilter7

Decodebin child added: h264parse9

Decodebin child added: capsfilter8

Decodebin child added: capsfilter9

Decodebin child added: nvv4l2decoder7

Decodebin child added: nvv4l2decoder8

Decodebin child added: nvv4l2decoder9

In cb_newpad

gstname= video/x-raw
sink_6
Decodebin linked to pipeline
In cb_newpad

gstname= video/x-raw
sink_0
Decodebin linked to pipeline
In cb_newpad

gstname= video/x-raw
sink_5
In cb_newpad

gstname= video/x-raw
sink_4
Decodebin linked to pipeline
Decodebin linked to pipeline
In cb_newpad

gstname= video/x-raw
sink_8
Decodebin linked to pipeline
In cb_newpad

gstname= video/x-raw
sink_3
Decodebin linked to pipeline
In cb_newpad

gstname= video/x-raw
sink_9
Decodebin linked to pipeline
In cb_newpad

gstname= video/x-raw
sink_1
Decodebin linked to pipeline
In cb_newpad

gstname= video/x-raw
sink_7
Decodebin linked to pipeline
In cb_newpad

gstname= video/x-raw
sink_2
Decodebin linked to pipeline

**PERF: {‘stream0’: 0.0, ‘stream1’: 0.0, ‘stream2’: 0.0, ‘stream3’: 0.0, ‘stream4’: 0.0, ‘stream5’: 0.0, ‘stream6’: 0.0, ‘stream7’: 0.0, ‘stream8’: 0.0, ‘stream9’: 0.0}

**PERF: {‘stream0’: 10.85, ‘stream1’: 10.85, ‘stream2’: 10.85, ‘stream3’: 10.85, ‘stream4’: 10.85, ‘stream5’: 10.85, ‘stream6’: 10.85, ‘stream7’: 10.85, ‘stream8’: 10.85, ‘stream9’: 10.85}

**PERF: {‘stream0’: 13.6, ‘stream1’: 13.6, ‘stream2’: 13.6, ‘stream3’: 13.6, ‘stream4’: 13.6, ‘stream5’: 13.6, ‘stream6’: 13.6, ‘stream7’: 13.6, ‘stream8’: 13.6, ‘stream9’: 13.6}

**PERF: {‘stream0’: 9.4, ‘stream1’: 9.4, ‘stream2’: 9.4, ‘stream3’: 9.4, ‘stream4’: 9.4, ‘stream5’: 9.4, ‘stream6’: 9.4, ‘stream7’: 9.4, ‘stream8’: 9.4, ‘stream9’: 9.4}

**PERF: {‘stream0’: 14.2, ‘stream1’: 14.2, ‘stream2’: 14.2, ‘stream3’: 14.2, ‘stream4’: 14.2, ‘stream5’: 14.2, ‘stream6’: 14.2, ‘stream7’: 14.2, ‘stream8’: 14.2, ‘stream9’: 14.2}

**PERF: {‘stream0’: 9.4, ‘stream1’: 9.4, ‘stream2’: 9.4, ‘stream3’: 9.4, ‘stream4’: 9.4, ‘stream5’: 9.4, ‘stream6’: 9.4, ‘stream7’: 9.4, ‘stream8’: 9.4, ‘stream9’: 9.4}

**PERF: {‘stream0’: 14.0, ‘stream1’: 14.0, ‘stream2’: 14.0, ‘stream3’: 14.0, ‘stream4’: 14.0, ‘stream5’: 14.0, ‘stream6’: 14.0, ‘stream7’: 14.0, ‘stream8’: 14.0, ‘stream9’: 14.0}

**PERF: {‘stream0’: 11.4, ‘stream1’: 11.4, ‘stream2’: 11.4, ‘stream3’: 11.4, ‘stream4’: 11.4, ‘stream5’: 11.4, ‘stream6’: 11.4, ‘stream7’: 11.4, ‘stream8’: 11.4, ‘stream9’: 11.4}

**PERF: {‘stream0’: 11.2, ‘stream1’: 11.2, ‘stream2’: 11.2, ‘stream3’: 11.2, ‘stream4’: 11.2, ‘stream5’: 11.2, ‘stream6’: 11.2, ‘stream7’: 11.2, ‘stream8’: 11.2, ‘stream9’: 11.2}

**PERF: {‘stream0’: 9.8, ‘stream1’: 9.8, ‘stream2’: 9.8, ‘stream3’: 9.8, ‘stream4’: 9.8, ‘stream5’: 9.8, ‘stream6’: 9.8, ‘stream7’: 9.8, ‘stream8’: 9.8, ‘stream9’: 9.8}

**PERF: {‘stream0’: 15.4, ‘stream1’: 15.4, ‘stream2’: 15.4, ‘stream3’: 15.4, ‘stream4’: 15.4, ‘stream5’: 15.4, ‘stream6’: 15.4, ‘stream7’: 15.4, ‘stream8’: 15.4, ‘stream9’: 15.4}

**PERF: {‘stream0’: 11.6, ‘stream1’: 11.6, ‘stream2’: 11.6, ‘stream3’: 11.6, ‘stream4’: 11.6, ‘stream5’: 11.6, ‘stream6’: 11.6, ‘stream7’: 11.6, ‘stream8’: 11.6, ‘stream9’: 11.6}

Calling Stop 1
STATE CHANGE SUCCESS

sink_1
STATE CHANGE SUCCESS

**PERF: {‘stream0’: 3.6, ‘stream1’: 3.6, ‘stream2’: 3.6, ‘stream3’: 3.6, ‘stream4’: 3.6, ‘stream5’: 3.6, ‘stream6’: 3.6, ‘stream7’: 3.6, ‘stream8’: 3.6, ‘stream9’: 3.6}

**PERF: {‘stream0’: 0.4, ‘stream1’: 0.2, ‘stream2’: 0.4, ‘stream3’: 0.4, ‘stream4’: 0.4, ‘stream5’: 0.4, ‘stream6’: 0.4, ‘stream7’: 0.4, ‘stream8’: 0.4, ‘stream9’: 0.2}
deepstream-test3.tar.gz (14.8 KB)

2 Likes
11

Hi Fiona,

It looks like when we set batch-push-timeout property of nvstreammux to a high value like here 4000000, nvstreammux causes the pipeline to stop when there is a change on the sinkpads. I believe it is probably that it has data from the stream in nvstreammux and it cannot handle it nicely while it no longer has the pad. We get lucky when we set batch-push-timeout to small value as nvstreammux has more chances to flush its buffers.

I believe nvstreammux cannot handle dynamic changes well here as seen on the example app. Could you please investigate this issue on the nvstreammux element?

Sending EOS event before unlinking, or starting flush before unlinking, or setting source element’s state to NULL doesn’t help. When nvstreammux gets these events it again halts the pipeline.

2 Likes
12

Can you upgrade to DeepStream 6.2? I have verified with DeepStream 6.2 with your modified app, it works.

13

Sure @Fiona.Chen , Let me try with DS 6.2, Do you think streammux has fixes around issue being reported in DS 6.2?

14

There are some bug fix patches with nvstreammux, maybe it is related to this issue.

15

Thank you @Fiona.Chen , Let me try it and update you.

16

Hello @Fiona.Chen ,

I have tried DS 6.2 and I can still see pipeline gets stuck randomly with number of sources. with DS 6.2, it stucks without dynamic addition/deletion of sources.

Attached are logs with GST_DEBUG=4 along with custom test app. I could regenerate same issue with /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/deepstream-test3/deepstream_test_3.py without any modification.

used below commands to test.
python3 deepstream_test_3.py -i rtsp://74.234.85.155:8554/live/stream1 rtsp://74.234.85.155:8554/live/stream2 rtsp://74.234.85.155:8554/live/stream3 rtsp://74.234.85.155:8554/live/stream4 rtsp://74.234.85.155:8554/live/stream5 rtsp://74.234.85.155:8554/live/stream6 rtsp://74.234.85.155:8554/live/stream7 rtsp://74.234.85.155:8554/live/stream8 rtsp://74.234.85.155:8554/live/stream9 rtsp://74.234.85.155:8554/live/stream10 -c config_infer_primary_peoplenet.txt -g nvinfer --no-display -s

ds6.2.log (1.2 MB)

deepstream_test_3.py (26.2 KB)

17

Hi @Fiona.Chen ,

I can reproduce this issue with DS 6.2 with lower timeout value also. it always stuck with 10 rtsp sources. I did not try lesser.

18

I can not reproduce with your app and 10 rtsp streams. Can you tell us which GPU you are using? Have you installed the DeepStream 6.2 as described in Quickstart Guide — DeepStream 6.2 Release documentation?

19

Hi Fiona,

I am using T4 GPU and I am using nvcr.io/nvidia/deepstream:6.2-triton container image and downloaded peoplenet model based on instruction given in README.

I have followed instruction given in Quickstart Guide — DeepStream 6.2 Release documentation. Nothing much done.

  1. Installed driver and build tool dependencies as describe in Quick Start guide.
  2. Installed docker, Nvidia-container tool kit and pulled docker image

image
image768×428 14.5 KB

Thanks,
Dilip Patel

20

Have you run “user_additional_install.sh” and “user_deepstream_python_apps_install.sh” in /opt/nvidia/deepstream/deepstream-6.2 to setup the pyds? Can the original python deepstream-test3 work in your docker? deepstream_python_apps/apps/deepstream-test3 at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

21

I have used ./user_deepstream_python_apps_install.sh --version 1.1.6 to setup pyds and deepstream.

I am seeing same issue with original python deepstream-test3 app in my container with 10 RTSP sources.

22

Hi @Fiona.Chen ,

I did setup my container again and tested with original python deepstream-test3 and it seems working now.

Do not know what has messed up yesteday’s test, let me test with my custom app also and verify.

Thanks for prompt response.