Exception when processing multiple sources with Deepstream

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
5.0
• JetPack Version (valid for Jetson only)
• TensorRT Version

• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

root@IDC_GPU_Server-1:/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app# deepstream-app -c test1.txt

 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8555/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8556/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8557/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8558/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8559/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8560/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8561/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8562/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8563/ds-test ***


 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8564/ds-test ***

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
WARNING: ../nvdsinfer/nvdsinfer_func_utils.cpp:36 [TRT]: Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
0:00:04.360260602 12915 0x5580f27ee4f0 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x368x640
1   OUTPUT kFLOAT conv2d_bbox     16x23x40
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40

0:00:04.360390895 12915 0x5580f27ee4f0 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine
0:00:04.370211802 12915 0x5580f27ee4f0 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/config_infer_primary.txt sucessfully
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency

Runtime commands:
        h: Print this help
        q: Quit

        p: Pause
        r: Resume

** INFO: <bus_callback:181>: Pipeline ready

** INFO: <bus_callback:167>: Pipeline running

KLT Tracker Init
KLT Tracker Init

**PERF:  FPS 0 (Avg)    FPS 1 (Avg)     FPS 2 (Avg)     FPS 3 (Avg)     FPS 4 (Avg)     FPS 5 (Avg)     FPS 6 (Avg)     FPS 7 (Avg)     FPS 8 (Avg)     FPS 9 (Avg)     FPS 10 (Avg)
**PERF:  0.00 (0.00)    0.00 (0.00)     0.00 (0.00)     0.00 (0.00)     0.00 (0.00)     23.24 (18.41)   0.00 (0.00)     0.00 (0.00)     0.00 (0.00)     19.35 (11.69)   0.00 (0.00)
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
** INFO: <bus_callback:167>: Pipeline running

KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
**PERF:  11.99 (11.62)  12.63 (12.44)   13.13 (12.91)   12.77 (12.56)   12.72 (12.57)   9.04 (9.30)     10.10 (9.98)    11.39 (11.24)   9.59 (9.48)     9.42 (9.44)     9.50 (9.39)
**PERF:  10.54 (11.16)  10.69 (11.50)   10.29 (11.40)   10.69 (11.51)   10.71 (11.58)   11.12 (10.13)   10.92 (10.40)   10.72 (10.90)   10.92 (10.14)   10.87 (10.01)   10.92 (10.11)
**PERF:  12.59 (11.68)  12.59 (11.90)   12.80 (11.85)   12.79 (11.99)   12.80 (11.95)   12.33 (10.95)   12.33 (11.15)   12.53 (11.49)   12.33 (10.96)   12.18 (10.87)   12.52 (11.02)
**PERF:  13.87 (9.29)   13.87 (9.41)    14.02 (9.33)    13.87 (9.44)    13.75 (9.54)    13.69 (8.88)    15.35 (8.98)    13.75 (9.24)    13.87 (8.87)    13.87 (8.81)    13.87 (8.89)
**PERF:  0.20 (7.40)    0.20 (7.47)     0.20 (7.37)     0.20 (7.47)     0.20 (7.59)     0.20 (7.15)     0.20 (7.20)     0.20 (7.37)     0.20 (7.13)     0.20 (7.09)     0.20 (7.14)
**PERF:  0.20 (6.16)    0.20 (6.20)     0.20 (6.11)     0.20 (6.19)     0.20 (6.31)     0.20 (6.00)     0.20 (6.02)     0.20 (6.15)     0.20 (5.97)     0.20 (5.95)     0.20 (5.97)
**PERF:  0.00 (5.26)    0.00 (5.28)     0.00 (5.19)     0.00 (5.26)     0.00 (5.39)     0.00 (5.15)     0.00 (5.16)     0.00 (5.25)     0.00 (5.12)     0.00 (5.10)     0.00 (5.12)
**PERF:  0.00 (4.58)    0.00 (4.60)     0.00 (4.51)     0.00 (4.58)     0.00 (4.69)     0.00 (4.51)     0.00 (4.51)     0.00 (4.58)     0.00 (4.48)     0.00 (4.46)     0.00 (4.47)
**PERF:  0.00 (4.06)    0.00 (4.07)     0.00 (3.99)     0.00 (4.05)     0.00 (4.16)     0.00 (4.01)     0.00 (4.00)     0.00 (4.07)     0.00 (3.98)     0.00 (3.97)     0.00 (3.97)
**PERF:  0.00 (3.65)    0.00 (3.65)     0.00 (3.58)     0.00 (3.63)     0.00 (3.74)     0.00 (3.61)     0.00 (3.60)     0.00 (3.65)     0.00 (3.58)     0.00 (3.57)     0.00 (3.57)
**PERF:  0.00 (3.31)    0.00 (3.31)     0.00 (3.24)     0.00 (3.29)     0.00 (3.39)     0.00 (3.28)     0.00 (3.27)     0.00 (3.32)     0.00 (3.26)     0.00 (3.25)     0.00 (3.25)
**PERF:  0.00 (3.03)    0.00 (3.03)     0.00 (2.97)     0.00 (3.01)     0.00 (3.10)     0.00 (3.01)     0.00 (3.00)     0.00 (3.04)     0.00 (2.99)     0.00 (2.98)     0.00 (2.98)

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
this is my config file

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=0
rows=1
columns=2
width=1280
height=720
gpu-id=0
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.7.123:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.6.163:554/h264/ch1/main/av_stream
#uri=rtsp://admin:infiled8899@10.10.150.112:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0


[source2]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.9.133:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source3]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.7.54:554/h264/ch1/main/av_stream
#uri=rtsp://admin:infiled8899@10.10.150.112:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source4]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.5.63:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source5]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.6.84:554/h264/ch1/main/av_stream
#uri=rtsp://admin:infiled8899@10.10.150.112:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source6]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.6.163:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source7]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.7.54:554/h264/ch1/main/av_stream
#uri=rtsp://admin:infiled8899@10.10.150.112:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0


[source8]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.5.63:554/h264/ch1/main/av_stream
#uri=rtsp://admin:infiled8899@10.10.150.112:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source9]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.6.84:554/h264/ch1/main/av_stream
#uri=rtsp://admin:infiled8899@10.10.150.112:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source10]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=rtsp://guest:Guest123@172.1.5.63:554/h264/ch1/main/av_stream
#uri=rtsp://admin:infiled8899@10.10.150.112:554/h264/ch1/main/av_stream
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=0
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400
gpu-id=0

[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8555
udp-port=5401
gpu-id=0

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=2
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8556
udp-port=5402
gpu-id=0

[sink3]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=3
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8557
udp-port=5403
gpu-id=0

[sink4]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8558
udp-port=5404
gpu-id=0

[sink5]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=5
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8559
udp-port=5405
gpu-id=0

[sink6]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=6
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8560
udp-port=5406
gpu-id=0

[sink7]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=7
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8561
udp-port=5407
gpu-id=0

[sink8]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=8
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8562
udp-port=5408
gpu-id=0

[sink9]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=9
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8563
udp-port=5409
gpu-id=0

[sink10]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
source-id=10
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8564
udp-port=5410
gpu-id=0

[osd]
enable=1
gpu-id=0
border-width=2
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=30
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=2560
height=1440
#width=1920
#height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine
batch-size=30
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tracker]
enable=1
# For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
#ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_mot_iou.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvdcf.so
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
#ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process and enable-past-frame applicable to DCF only
enable-batch-process=0
enable-past-frame=0
display-tracking-id=1

[tests]
file-loop=0

Please upgrade to latest Deepstream 6.0 GA version. The old version is not supported now.

In [streammux], please set “batch-size=11”, you have only 11 sources.

You can set “GST_DEBUG=3” to get more log for debugging.