Dynamic sensor removal in gst-nvmultiurisrcbin resulting in hanging of pipeline

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) → GPU
• DeepStream Version → 6.4 (nvcr.io/nvidia/deepstream:6.4-gc-triton-devel)
• NVIDIA GPU Driver Version (valid for GPU only) → 535.154.05
• Issue Type( questions, new requirements, bugs) → Bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
When using the gst-nvmultiurisrcbin in the pipeline, while testing the Dynamic addition/removal of sources I have encountered the pipeline being hung or throwing segmentation fault, when trying to remove the source which has been added. Here are the steps and logs related to the issue.

## Docker container had been built with glib 2.79.1 using the commands below
FROM nvcr.io/nvidia/deepstream:6.4-triton-multiarch

RUN DEBIAN_FRONTEND=noninteractive apt-get update -y
RUN DEBIAN_FRONTEND=noninteractive apt-get install ca-certificates
RUN DEBIAN_FRONTEND=noninteractive apt-get update -y

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y sudo apt-utils make g++ build-essential wget unzip \
              libprocps-dev libjson-c-dev libcurl4-openssl-dev openssh-client git cmake sqlite3 \
              libsqlite3-dev libi2c-dev libopenblas-base libopenmpi-dev libnuma-dev libopenblas-dev \
              libsasl2-modules-gssapi-mit libsasl2-dev openssl libssl-dev telnet telnet-ssl aptly openssh-server locales \
              pkg-config python3-dev libffi-dev libmount-dev ninja-build libgirepository1.0-dev netcat

RUN locale-gen en_US.UTF-8

RUN pip3 install packaging
RUN pip3 install meson==1.2.0
RUN wget https://github.com/GNOME/glib/archive/refs/tags/2.79.1.tar.gz
RUN tar -xzvf ./2.79.1.tar.gz
WORKDIR glib-2.79.1
RUN ls -l
RUN rm -r subprojects/gvdb
RUN meson _build
RUN ninja -C _build
RUN ninja -C _build install

RUN rsync -av /usr/local/lib/x86_64-linux-gnu/ /usr/lib/x86_64-linux-gnu/

The above created image has been used to test the pipeline and here is the docker run command used

docker run -it -d --network host --runtime nvidia -v /tmp/.X11-unix:/tmp/.X11-unix     -e DISPLAY=$DISPLAY --volume ${PWD}:/root/apps/ -w /root/apps/ --privileged    deepstream_dev bash

Inside the container the below pipeline is executed

export USE_NEW_NVSTREAMMUX=yes
gst-launch-1.0 nvmultiurisrcbin \
port=9000 ip-address=localhost width=1920 height=1080 \
batched-push-timeout=33333 sync-inputs=1 config-file-path=../config/new_nvstreammux.cfg \
drop-pipeline-eos=1 live-source=1 \
uri-list=rtsp://127.0.0.1:9564/cam10 \
! fakesink async=0

Adding of the source is done using: http://172.16.10.97:9000/api/v1/stream/add

{
    "key": "sensor",
    "value": {
        "camera_id": "test",
        "camera_url": "rtsp://127.0.0.1:9565/cam11",
        "change": "camera_add"
    }
}

Removal of the same source by using: http://172.16.10.97:9000/api/v1/stream/remove

{
    "key": "sensor",
    "value": {
        "camera_id": "test",
        "camera_url": "rtsp://127.0.0.1:9565/cam11",
        "change": "camera_remove"
    }
}

Here are the logs of the pipeline

uri:/api/v1/stream/remove2 %)
method:POST
Progress: (request) Sending PAUSE request
Progress: (request) Sending PAUSE request
Progress: (request) Sent PAUSE request
Progress: (close) Closing Stream
Progress: (close) Closing stream
Progress: (close) Closed Stream
Caught SIGSEGV
#0  0x00007f05f9ff2dbf in poll () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f05fa1c1e7f in g_main_context_poll_unlocked (priority=<optimized out>, n_fds=2, fds=0x560db01d74e0, timeout=<optimized out>, context=0x560dafb800a0) at ../glib/gmain.c:4521
#2  g_main_context_iterate_unlocked (context=0x560dafb800a0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4212
#3  0x00007f05fa1c2743 in g_main_loop_run (loop=0x560dafb583e0) at ../glib/gmain.c:4419
#4  0x0000560dadb45adb in ?? ()
#5  0x00007f05f9f03d90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007f05f9f03e40 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000560dadb45c05 in _start ()
Spinning.  Please run 'gdb gst-launch-1.0 31671' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
0:00:24.4 / 0:10:00.6 (4.1 %)

## Backtrace from gdb
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f05f9ff2dbf in poll () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) backtrace 
#0  0x00007f05f9ff2dbf in poll () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f05fa1c1e7f in g_main_context_poll_unlocked (priority=<optimized out>, n_fds=2, 
    fds=0x560db01d74e0, timeout=<optimized out>, context=0x560dafb800a0) at ../glib/gmain.c:4521
#2  g_main_context_iterate_unlocked (context=0x560dafb800a0, block=block@entry=1, 
    dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4212
#3  0x00007f05fa1c2743 in g_main_loop_run (loop=0x560dafb583e0) at ../glib/gmain.c:4419
#4  0x0000560dadb45adb in ?? ()
#5  0x00007f05f9f03d90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007f05f9f03e40 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000560dadb45c05 in _start ()
3

Thanks for filing this problem. We will analyze this as soon as possible.