Slow decoding T4 DS 5.0.1

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 5.0.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 455.45.01
• Issue Type( questions, new requirements, bugs)

I am using deepstream-test5 with 15x1080p rtsp cameras all at 10fps. I have measured the latency and get the following:

************BATCH-NUM = 12191**************
Comp name = nvv4l2decoder9 in_system_timestamp = 1614650126597.912109 out_system_timestamp = 1614650126672.937012               component latency= 75.024902
Comp name = src_bin_muxer source_id = 14 pad_index = 14 frame_num = 11477               in_system_timestamp = 1614650126672.982910 out_system_timestamp = 1614650126868.168945               component_latency = 195.186035
Comp name = nvv4l2decoder11 in_system_timestamp = 1614650126594.323975 out_system_timestamp = 1614650126720.464111               component latency= 126.140137
Comp name = src_bin_muxer source_id = 8 pad_index = 8 frame_num = 11477               in_system_timestamp = 1614650126720.522949 out_system_timestamp = 1614650126868.168945               component_latency = 147.645996
Comp name = nvv4l2decoder0 in_system_timestamp = 1614650126593.139893 out_system_timestamp = 1614650126711.650879               component latency= 118.510986
Comp name = src_bin_muxer source_id = 9 pad_index = 9 frame_num = 11478               in_system_timestamp = 1614650126711.681885 out_system_timestamp = 1614650126868.168945               component_latency = 156.487061
Comp name = nvv4l2decoder2 in_system_timestamp = 1614650126695.525879 out_system_timestamp = 1614650126781.639893               component latency= 86.114014
Comp name = src_bin_muxer source_id = 2 pad_index = 2 frame_num = 11479               in_system_timestamp = 1614650126781.718018 out_system_timestamp = 1614650126868.168945               component_latency = 86.450928
Comp name = nvv4l2decoder12 in_system_timestamp = 1614650126699.129883 out_system_timestamp = 1614650126782.073975               component latency= 82.944092
Comp name = src_bin_muxer source_id = 3 pad_index = 3 frame_num = 11477               in_system_timestamp = 1614650126782.115967 out_system_timestamp = 1614650126868.169922               component_latency = 86.053955
Comp name = nvv4l2decoder13 in_system_timestamp = 1614650126709.569092 out_system_timestamp = 1614650126784.552979               component latency= 74.983887
Comp name = src_bin_muxer source_id = 11 pad_index = 11 frame_num = 11477               in_system_timestamp = 1614650126784.611084 out_system_timestamp = 1614650126868.169922               component_latency = 83.558838
Comp name = nvv4l2decoder8 in_system_timestamp = 1614650126677.824951 out_system_timestamp = 1614650126799.634033               component latency= 121.809082
Comp name = src_bin_muxer source_id = 4 pad_index = 4 frame_num = 11478               in_system_timestamp = 1614650126799.675049 out_system_timestamp = 1614650126868.169922               component_latency = 68.494873
Comp name = nvv4l2decoder1 in_system_timestamp = 1614650126709.485107 out_system_timestamp = 1614650126805.604980               component latency= 96.119873
Comp name = src_bin_muxer source_id = 7 pad_index = 7 frame_num = 11479               in_system_timestamp = 1614650126805.687012 out_system_timestamp = 1614650126868.169922               component_latency = 62.482910
Comp name = nvv4l2decoder4 in_system_timestamp = 1614650126739.525879 out_system_timestamp = 1614650126817.781982               component latency= 78.256104
Comp name = src_bin_muxer source_id = 1 pad_index = 1 frame_num = 11479               in_system_timestamp = 1614650126817.827881 out_system_timestamp = 1614650126868.169922               component_latency = 50.342041
Comp name = nvv4l2decoder3 in_system_timestamp = 1614650126741.389893 out_system_timestamp = 1614650126818.105957               component latency= 76.716064
Comp name = src_bin_muxer source_id = 6 pad_index = 6 frame_num = 11479               in_system_timestamp = 1614650126818.144043 out_system_timestamp = 1614650126868.170898               component_latency = 50.026855
Comp name = nvv4l2decoder10 in_system_timestamp = 1614650126712.000000 out_system_timestamp = 1614650126835.852051               component latency= 123.852051
Comp name = src_bin_muxer source_id = 5 pad_index = 5 frame_num = 11478               in_system_timestamp = 1614650126835.990967 out_system_timestamp = 1614650126868.170898               component_latency = 32.179932
Comp name = primary_gie in_system_timestamp = 1614650126868.590088 out_system_timestamp = 1614650126891.912109               component latency= 23.322021
Comp name = tracking_tracker in_system_timestamp = 1614650126891.919922 out_system_timestamp = 1614650126914.053955               component latency= 22.134033
Comp name = dsanalytics0 in_system_timestamp = 1614650126914.095947 out_system_timestamp = 1614650126915.117920               component latency= 1.021973
Comp name = tiled_display_tiler in_system_timestamp = 1614650126915.351074 out_system_timestamp = 1614650126916.775879               component latency= 1.424805
Comp name = osd_conv in_system_timestamp = 1614650126917.218994 out_system_timestamp = 1614650126918.149902               component latency= 0.930908
Comp name = nvosd0 in_system_timestamp = 1614650126927.258057 out_system_timestamp = 1614650126940.207031               component latency= 12.948975
Source id = 14 Frame_num = 11477 Frame latency = 342.425781 (ms) 
Source id = 8 Frame_num = 11477 Frame latency = 346.013916 (ms) 
Source id = 9 Frame_num = 11478 Frame latency = 347.197998 (ms) 
Source id = 2 Frame_num = 11479 Frame latency = 244.812012 (ms) 
Source id = 3 Frame_num = 11477 Frame latency = 241.208008 (ms) 
Source id = 11 Frame_num = 11477 Frame latency = 230.768799 (ms) 
Source id = 4 Frame_num = 11478 Frame latency = 262.512939 (ms) 
Source id = 7 Frame_num = 11479 Frame latency = 230.852783 (ms) 
Source id = 1 Frame_num = 11479 Frame latency = 200.812012 (ms) 
Source id = 6 Frame_num = 11479 Frame latency = 198.947998 (ms) 
Source id = 5 Frame_num = 11478 Frame latency = 228.337891 (ms)  

The decoder and muxer appear to be taking the majority of the time, much greater than expected. After a few hours the system is struggling and dropping frames. The DS config is:



[application]
enable-perf-measurement=1
perf-measurement-interval-sec=60
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=4
columns=4
width=1920
height=1080
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.90/axis-media/media.amp
gpu-id=0
rtsp-reconnect-interval-sec=600
cudadec-memtype=0

[source1]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.87/axis-media/media.amp
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source2]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.88/axis-media/media.amp
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source3]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.76/axis-media/media.amp
rtsp-reconnect-interval-sec=600000
gpu-id=0
cudadec-memtype=0

[source4]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.73/axis-media/media.amp?camera=1
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source5]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.65/axis-media/media.amp
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source6]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.93/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source7]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.95/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source8]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.94/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source9]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.97/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source10]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.98/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source11]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.86/axis-media/media.amp?camera=1
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source12]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.100/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source13]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.99/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source14]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.96/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
qos=0
source-id=0
gpu-id=0
nvbuf-memory-type=0
#1=mp4 2=mkv
#container=1
#1=h264 2=h265
#codec=1
#output-file=yolov4.mp4

#DEPLOYMENT WITH AZURE
[sink1]
enable=1
sync=0
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=naming_conv_iotedge.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
msg-conv-payload-type=1
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream/lib/libnvds_azure_edge_proto.so
#msg-broker-conn-str=localhost;5672;guest;guest
topic=mytopic

#RTSP output to see analytics
[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=4
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0
codec=2
#enc-type=0
bitrate=4000000
iframeinterval=20
rtsp-port=8554
udp-port=5000
profile=2
udp-buffer-size=4000000

[osd]
enable=1
gpu-id=0
display-text=1
border-width=2
process-mode=1
text-size=10
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=15
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=100000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0

[primary-gie]
enable=1
gpu-id=0
labelfile-path=labels.txt
batch-size=15
force-implicit-batch-dim=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=2
gie-unique-id=1
filter-out-class-ids=25
operate-on-class-ids=0
nvbuf-memory-type=0
#config-file=config_infer_primary_yoloV4.txt
config-file=config_infer_primary_mish.txt

[tracker]
enable=1
tracker-width=640
tracker-height=640
gpu-id=0
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
ll-config-file=tracker_config.yml
enable-batch-process=1
#enable-past-frame=1

[nvds-analytics]
enable=1
config-file=analytics_line.txt

I have read all docs and set streamux @ camera resolution, all sinks syc=0, latency on sources etc.

3

Can you show your deepstream-app config file?

What is the GPU loading of this case?

4 
[application]
enable-perf-measurement=1
perf-measurement-interval-sec=60
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=4
columns=4
width=1920
height=1080
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.90/axis-media/media.amp
gpu-id=0
rtsp-reconnect-interval-sec=600
cudadec-memtype=0

[source1]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.87/axis-media/media.amp
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source2]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.88/axis-media/media.amp
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source3]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.76/axis-media/media.amp
rtsp-reconnect-interval-sec=600000
gpu-id=0
cudadec-memtype=0

[source4]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.73/axis-media/media.amp?camera=1
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source5]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.65/axis-media/media.amp
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source6]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.93/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source7]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.95/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source8]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.94/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source9]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.97/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source10]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.98/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source11]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://root:pass@192.168.100.86/axis-media/media.amp?camera=1
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source12]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.100/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source13]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.99/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[source14]
enable=1
latency=1000
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=4
uri=rtsp://admin:pass@192.168.100.96/Streaming/Channels/101
rtsp-reconnect-interval-sec=600
gpu-id=0
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
qos=0
source-id=0
gpu-id=0
nvbuf-memory-type=0
#1=mp4 2=mkv
#container=1
#1=h264 2=h265
#codec=1
#output-file=yolov4.mp4

#DEPLOYMENT WITH AZURE
[sink1]
enable=1
sync=0
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=naming_conv_iotedge.txt
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
msg-conv-payload-type=1
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream/lib/libnvds_azure_edge_proto.so
#msg-broker-conn-str=localhost;5672;guest;guest
topic=mytopic

#RTSP output to see analytics
[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=4
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0
codec=2
#enc-type=0
bitrate=4000000
iframeinterval=20
rtsp-port=8554
udp-port=5000
profile=2
udp-buffer-size=4000000

[osd]
enable=1
gpu-id=0
display-text=1
border-width=2
process-mode=1
text-size=10
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=15
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=100000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0

[primary-gie]
enable=1
gpu-id=0
labelfile-path=labels.txt
batch-size=15
force-implicit-batch-dim=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=2
gie-unique-id=1
filter-out-class-ids=25
operate-on-class-ids=0
nvbuf-memory-type=0
#config-file=config_infer_primary_yoloV4.txt
config-file=config_infer_primary_mish.txt

[tracker]
enable=1
tracker-width=640
tracker-height=640
gpu-id=0
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
ll-config-file=tracker_config.yml
enable-batch-process=1
#enable-past-frame=1

[nvds-analytics]
enable=1
config-file=analytics_line.txt

GPU Loading:

Tue Mar  2 15:31:23 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01    Driver Version: 455.45.01    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:3B:00.0 Off |                    0 |
| N/A   69C    P0    37W /  70W |   3437MiB / 15109MiB |     69%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
5

Can you use “nvidia-smi dmon” to show the GPU loading details for running the case?

6 
# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0    67    68     -    45    18     6     9  5000  1395
    0    94    69     -    76    30     6     9  5000  1005
    0    36    69     -    44    19     5     9  5000  1515
    0    71    69     -    63    24     6     9  5000  1005
    0    37    68     -    56    25     5     9  5000  1575
    0    56    68     -    40    17     5     8  5000  1500
    0    95    69     -    69    27     1     9  5000  1005
    0    35    68     -    44    20     5     7  5000  1545
    0    72    69     -    56    23     0    10  5000  1050
    0    60    70     -    75    32     5     8  5000  1275
    0    35    68     -    44    18     5     7  5000  1500
    0   101    69     -    66    25     0     9  5000  1005
    0    36    68     -    47    23     5     9  5000  1575
    0    67    68     -    51    21     5    10  5000  1395
    0    41    69     -    80    31     5    10  5000  1155
    0    35    69     -    39    17     5     8  5000  1560
    0    76    69     -    68    26     0     9  5000  1050
    0    50    69     -    66    28     5     7  5000  1380
    0    35    68     -    42    18     5     7  5000  1425
    0    93    69     -    71    27     4    10  5000  1005
    0    49    69     -    59    26     5     7  5000  1320
    0    42    68     -    47    20     5     9  5000  1515
    0    92    69     -    61    23     0    10  5000  1020
    0    42    68     -    55    25     5     8  5000  1575
    0    65    68     -    48    20     2     9  5000  1440
    0    91    69     -    64    29     5    10  5000  1020
7

What is your camera’s fps?

8

15x1080p rtsp cameras all at 10fps

9

Do you mean the your ip camera generate 10 fps streams with 1080p resolution?

10

There are 15 different rtsp cameras, all of them are set at 1080p resoltion and 10fps

11

The latency of each component is the buffer in and out duration but not module processing time. Can you try to enlarge the “latency” setting in [source] group to try? E.G. “latency=4000”.

12

Will try and see if it helps