Performance drop in multi source rtsp stream processing

hrsht.sarma · April 8, 2020, 11:59am

I am facing unexpected behavior while processing multiple rtsp streams on the same dGPU. Initially when I was using 2 streams it was working fine , now when I added 2 more streams , after sometime one source FPS drops to 0. Sometimes certain source’s FPS drops to 0 , recovers and then drops to 0 again .

 **PERF: FPS 0 (Avg)	FPS 1 (Avg)	FPS 2 (Avg)	FPS 3 (Avg)	
**PERF: 6.78 (6.78)	0.00 (0.00)	0.00 (0.00)	15.47 (15.47)	
**PERF: 6.01 (6.07)	0.00 (0.00)	0.00 (0.00)	11.80 (12.15)	
**PERF: 6.01 (6.04)	13.30 (13.30)	3.80 (3.80)	12.02 (12.08)	
**PERF: 5.88 (6.03)	6.08 (6.48)	0.37 (0.68)	11.98 (12.05)	
**PERF: 0.00 (6.03)	11.85 (8.34)	6.87 (4.37)	12.02 (12.04)	
**PERF: 0.00 (6.03)	11.99 (9.27)	11.01 (5.56)	11.99 (12.03)	
**PERF: 0.00 (6.03)	11.37 (9.70)	4.98 (5.40)	11.77 (11.99)	
**PERF: 0.00 (6.03)	11.39 (9.98)	11.19 (6.36)	12.00 (11.99)	
**PERF: 0.00 (6.03)	12.02 (10.28)	7.23 (6.47)	12.02 (12.00)	
**PERF: 0.00 (6.03)	11.80 (10.47)	2.43 (6.29)	12.00 (12.00)	
**PERF: 0.00 (6.03)	11.19 (10.55)	3.66 (5.76)	11.59 (11.96)	
**PERF: 0.00 (6.03)	11.22 (10.62)	9.92 (6.07)	11.62 (11.93)	
**PERF: 0.00 (6.03)	11.66 (10.71)	1.80 (5.75)	11.57 (11.90)`

hrsht.sarma · April 9, 2020, 6:42am

Even today I am facing similar issue:

**PERF: FPS 0 (Avg)	FPS 1 (Avg)	FPS 2 (Avg)	FPS 3 (Avg)	
**PERF: 6.20 (6.20)	0.00 (0.00)	0.00 (0.00)	13.76 (13.76)	
**PERF: 5.64 (5.77)	0.00 (0.00)	0.00 (0.00)	11.33 (11.87)	
**PERF: 5.62 (5.74)	5.46 (5.46)	5.86 (5.86)	11.41 (11.67)	
**PERF: 0.00 (5.74)	11.44 (7.84)	11.44 (8.07)	11.64 (11.66)	
**PERF: 0.00 (5.74)	11.37 (8.82)	11.37 (8.99)	11.37 (11.59)	
**PERF: 0.00 (5.74)	11.52 (9.42)	11.52 (9.55)	11.52 (11.58)	
**PERF: 0.00 (5.74)	11.22 (9.75)	4.33 (8.63)	11.22 (11.52)	
**PERF: 0.00 (5.74)	11.35 (9.99)	7.93 (8.52)	11.35 (11.50)	
**PERF: 0.00 (5.74)	11.24 (10.16)	10.05 (8.73)	11.24 (11.47)	
**PERF: 0.00 (5.74)	11.61 (10.32)	11.61 (9.06)	11.61 (11.48)	
**PERF: 0.00 (5.74)	11.40 (10.44)	11.20 (9.29)	11.20 (11.45)

Amycao · April 9, 2020, 1:50pm

Hi
You can enable NVDS_ENABLE_LATENCY_MEASUREMENT and NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT to see components latency, if your rtsp camera or stream have network latency, it will impact decoder and streammux latency, and streammux property live set to 1 will have improvements, also set sink sync to 0;
we have jitterbuffer property set which will improve the situation in upcoming release.

hrsht.sarma · April 10, 2020, 2:10pm

set sync=0 in streammux? setting live-source=1 is not much of a relief here.

hrsht.sarma · April 11, 2020, 9:59am

Any other way to fix this , if there is none then I can move on and try something else.

Amycao · April 13, 2020, 1:06am

set sync=0 in sink

hrsht.sarma · June 3, 2020, 8:43am

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=../../../kitti/
kitti-track-output-dir=../../../kitti_track/

[tiled-display]
enable=0
rows=1
columns=1
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
#uri=file://../../../streams/sample_1080p_h264.mp4
#uri=file://../../../../../harshit_sample/harshit_sample_vantagedorm_1.mp4
uri=rtsp://admin:@192.168.1.10:554/mode=real&idc=1&ids=1
#uri=file://../../../../../covid/2020-05-09_17_45.h264
num-sources=1
gpu-id=0
drop-frame-interval=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
#uri=file://../../../streams/sample_1080p_h264.mp4
uri=rtsp://admin:@192.168.1.12:554/mode=real&idc=1&ids=1
num-sources=1
drop-frame-interval=0
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0

[sink1]
enable=1
type=1
sync=0
source-id=1
gpu-id=0
nvbuf-memory-type=0

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=2
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=2560
height=1440
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../../models/Primary_Detector/resnet10.caffemodel_b4_int8.engine
batch-size=4
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tracker]
enable=1
tracker-width=640
tracker-height=368
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_nvdcf.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1

[secondary-gie]
enable=1
model-engine-file=../../../models/Secondary_FaceDetect/fd_lpd_model/fd_lpd.caffemodel_b1_fp32.engine
gpu-id=0
batch-size=4
gie-unique-id=4
operate-on-gie-id=1
#operate-on-class-ids=0;
config-file=config_infer_secondary.txt

[ds-example]
enable=1
processing-width=2560
processing-height=1440
full-frame=0
unique-id=15
gpu-id=0

[tests]
file-loop=0

When I add more than 1 source I dont get any FPS:

Creating LL OSD context new

**PERF: FPS 0 (Avg)     FPS 1 (Avg)
**PERF: 0.00 (0.00)     0.00 (0.00)
Creating LL OSD context new
** INFO: <bus_callback:175>: Pipeline running

**PERF: 0.00 (0.00)     0.00 (0.00)
**PERF: 0.00 (0.00)     0.00 (0.00)
**PERF: 0.00 (0.00)     0.00 (0.00)
**PERF: 0.00 (0.00)     0.00 (0.00)
**PERF: 0.00 (0.00)     0.00 (0.00)
**PERF: 0.00 (0.00)     0.00 (0.00)

hrsht.sarma · June 5, 2020, 7:31am

This issue still persist

Amycao · June 10, 2020, 7:57am

May I know which platform you working on?

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)

please check the troubleshoot, part The DeepStream application is running slowly.

https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_troubleshooting.html#

kayccc · June 23, 2020, 1:30am

I’m closing this topic due to there is no update from you for a period, assuming this issue was resolved.
If still need the support, please open a new topic. Thanks