Running multiple RTSP streams causes buffer loop

Hi,

we’re running into an issue where a set of buffers is looped indefinitely for a specific RSTP stream.

it doesn’t happen immediately, nor are there any messages posted when it does happen.
unfortunately trying to run it with GST_DEBUG level 5 goes too slow for a good test to take place

i’ve been adding signal and message handlers to look for GstElement activity that could help track down the cause.

the setup is…
deepstream custom app running in NVIDIA deepstream container (also used demo deepstream-app)
one instance per RTSP stream (only one stream per app instance, one app instance per container instance)
config is set to have batch size of 1 for mux and primary GIE
config is set for source latency to be 0 (same issue with latency of 100 and 200)
sources are live
sources tried - Axis q1785, VLC rtsp server

when the loop happens with an object that can be inferenced, the track ID increases which indicates the problem is before tracking.

i’ve modified the gstreamer queues to be have more capacity, and also made them leaky downstream to avoid queueing issues, still get the same result.
i.e.
g_object_set(G_OBJECT(bin->dec_que), “leaky”, 2, NULL);

g_object_set(G_OBJECT(bin->dec_que), "max-size-buffers", 50, NULL);

have seen the same problem exist for both the demo deepstream-app and my custom app

does anyone have any ideas of where I should be looking to investigate this buffer looping?
i’ve added signal callbacks for the queues and they rarely get above 1 buffer in the queue
i was hoping to see an overrun on one of them.

any ideas on what else can be interrogated?

there is no queue between primary GIE and tracker so my guess is the issue is somewhere between source and tracker, downstream elements act as if they receive new data

Thanks,

Rich

Update…

disabling primary GIE causes the issue to go away.
no looped buffers

Is there anything in nvvidconv or nvinfer to check?

Hi

I still don’t clear the issue “a set of buffers is looped indefinitely for a specific RSTP stream”
Can I reproduce on my side? Can you share vlc command to create rtsp server and your code ?

HI Chris,

vlc server command was…

cvlc --loop test-carpark.mkv :sout=#gather:rtp{sdp=rtsp://:8654/file-stream} :network-caching=1500 :sout-all :sout-keep

the issue is that after a while the deepstream application reprocesses the same batch of frames over and over, rather than new frames from the source.

visually this appears like a short gif set to repeat, however the inferenced objects get new track ids as per the trackers usual behaviour.

we have the same issue when processing large video files as well.

i’ve run the process with GST_DEBUG for nvinfer and nvvidconv set to 5 but that doesn’t show much at all.

log.txt (995 KB)

Did you verify the source by vlc client playing?

in the log file attached to the previous message you can see normal behaviour up until line 2820
from there the issue appears, which in this instance shows a regular repeat of two inferences that meet our conditions for export and are printed to the terminal.

e.g. frametimestamp is: 12345678900000000 - framenumber is 123456789

yes could play the RTSP source from a different vlc window
and could see the output from the DS rtsp stream as well

Hi
I run the default deepstream-app and the below config about 10 minutues, but there is no the issue “the same batch of frames over and over, rather than new frames from the source.”

# Copyright (c) 2018 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0
# 0 - cuda pinned/host memory
# 1 - cuda device memory
# 2 - cuda unified memory
cuda-memory-type=1

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
#uri=file://../../streams/sample_720p.mp4
uri=rtsp://10.23.144.246:8654/file-stream
num-sources=1
gpu-id=0
# 0 - cuda pinned/host memory
# 1 - cuda device memory
# 2 - cuda unified memory
cuda-memory-type=1

[source1]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file://../../streams/sample_720p.mp4
num-sources=15
gpu-id=0
cuda-memory-type=1

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=0
gpu-id=0
cuda-memory-type=1

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265 3=mpeg4
## only SW mpeg4 is supported right now.
codec=3
sync=1
bitrate=2000000
output-file=out.mp4
source-id=0

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
cuda-memory-type=1

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1280
height=720
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
cuda-memory-type=1

[primary-gie]
enable=1
gpu-id=0
batch-size=30
## 0=FP32, 1=INT8, 2=FP16 mode
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;1;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
cuda-memory-type=1
config-file=config_infer_primary.txt
labelfile-path=../../Model/IVAPrimary_resnet10_DeepstreamRel_V2_ivalarge_its_phase1/labels.txt
model-engine-file=../../Model/IVAPrimary_resnet10_DeepstreamRel_V2_ivalarge_its_phase1/resnet10.caffemodel_b30_int8.engine
#infer-raw-output-dir=../../primary_detector_raw_output/

[ds-example]
enable=0
gpu-id=0
processing-width=640
processing-height=360
full-frame=1
cuda-memory-type=1

[tests]
file-loop=1

Hi Chris,

here is the config used…

# Copyright (c) 2018 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.


[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1920
height=1080
gpu-id=0
# 0 - cuda pinned/host memory
# 1 - cuda device memory
# 2 - cuda unified memory
cuda-memory-type=1

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
#uri=file:///root/DeepStream_Release/samples/streams/sample_720p.mp4
#uri=file://../../brisbane.avi
#uri=rtsp://root:DigExp2907@192.168.1.11/axis-media/media.amp
#uri=file:///root/videos/2019-05-01/11/30_04_2019 11_59_59 PM (UTC+10_00).mkv
uri=file:///root/sample-videos/lane-1-front-20190505.mkv
num-sources=1
gpu-id=0
# 0 - cuda pinned/host memory
# 1 - cuda device memory
# 2 - cuda unified memory
cuda-memory-type=1

[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=1
source-id=0
gpu-id=0
cuda-memory-type=1

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265 3=mpeg4
## only SW mpeg4 is supported right now.
codec=3
sync=0
bitrate=2000000
output-file=out.mp4
source-id=0

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=mp4 2=mkv
container=2
#1=h264 2=h265 3=mpeg4
## only mpeg4 is supported right now.
codec=1
sync=0
bitrate=4000
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5405


[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
cuda-memory-type=1

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
#batch-size=4
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
cuda-memory-type=1

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=/root/DeepStream_Release/samples/models/Primary_Detector/resnet10.caffemodel_b4_int8.engine
labelfile-path=/root/DeepStream_Release/samples/models/Primary_Detector/labels.txt
batch-size=4
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=2
gie-unique-id=1
cuda-memory-type=1
config-file=/root/DeepStream_Release/samples/configs/deepstream-app/config_infer_primary.txt

[tracker]
enable=1
tracker-width=640
tracker-height=368
gpu-id=0

[secondary-gie0]
enable=0
model-engine-file=/root/DeepStream_Release/samples/models/Secondary_VehicleTypes/resnet18.caffemodel_b16_int8.engine
labelfile-path=/root/DeepStream_Release/samples/models/Secondary_VehicleTypes/labels.txt
gpu-id=0
batch-size=16
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=/root/DeepStream_Release/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt

[secondary-gie1]
enable=0
model-engine-file=/root/DeepStream_Release/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_int8.engine
labelfile-path=/root/DeepStream_Release/samples/models/Secondary_CarColor/labels.txt
batch-size=16
gpu-id=0
gie-unique-id=5
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=/root/DeepStream_Release/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt

[secondary-gie2]
enable=0
model-engine-file=/root/DeepStream_Release/samples/models/Secondary_CarMake/resnet18.caffemodel_b16_int8.engine
labelfile-path=/root/DeepStream_Release/samples/models/Secondary_CarMake/labels.txt
batch-size=16
gpu-id=0
gie-unique-id=6
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=/root/DeepStream_Release/samples/configs/deepstream-app/config_infer_secondary_carmake.txt

[tests]
file-loop=1

[vehcio]
enable=1
processing-width=640
processing-height=480
full-frame=0
unique-id=3
gpu-id=0
config-url=http://172.17.0.2/api/v1/3/shapes/
redis-host=172.17.0.2

Hi Chris,

I’ve just run the app through cuda-memcheck and found a few errors.
see attached for the full log

have you seen anything like this before?

========= Invalid __global__ write of size 1
=========     at 0x00000278 in drawBoundingBox_kernel(uchar4*, int, int, int, int, int, int, int, unsigned int, unsigned int, unsigned int, unsigned int)
=========     by thread (15,8,0) in block (7,67,0)
=========     Address 0x7f5cdcfe91fe is out of bounds
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 (cuLaunchKernel + 0x2cd) [0x24f9bd]
=========     Host Frame:/usr/local/cuda-10.0/lib64/libcudart.so.10.0 [0x1a23d]
=========     Host Frame:/usr/local/cuda-10.0/lib64/libcudart.so.10.0 [0x1a2c7]
=========     Host Frame:/usr/local/cuda-10.0/lib64/libcudart.so.10.0 (cudaLaunchKernel + 0x225) [0x4e3c5]
=========     Host Frame:///usr/local/deepstream/libnvds_osd.so [0x5c07]
=========     Host Frame:///usr/local/deepstream/libnvds_osd.so (_Z59__device_stub__Z22drawBoundingBox_kernelP6uchar4iiiiiiijjjjP6uchar4iiiiiiijjjj + 0x22c) [0x5341]
=========     Host Frame:///usr/local/deepstream/libnvds_osd.so (_Z22drawBoundingBox_kernelP6uchar4iiiiiiijjjj + 0x52) [0x53a3]
=========     Host Frame:///usr/local/deepstream/libnvds_osd.so (_Z20drawBoundingBox_cudaPhiiiiiiiP11CUstream_stjjjj + 0xe0) [0x4cd0]
=========     Host Frame:///usr/local/deepstream/libnvds_osd.so [0x2a9c]
=========     Host Frame:///usr/local/deepstream/libnvds_osd.so (nvosd_draw_rectangles + 0x88) [0x2d2c]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libnvdsgst_osd.so [0x3120]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0 [0x35c66]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0 [0x35446]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x6e59f]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (gst_pad_push + 0x113) [0x76543]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0 [0x3552d]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x6e59f]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (gst_pad_push + 0x113) [0x76543]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstcoreelements.so [0x2ca69]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0xa0f31]
=========     Host Frame:/lib/x86_64-linux-gnu/libglib-2.0.so.0 [0x715ee]
=========     Host Frame:/lib/x86_64-linux-gnu/libglib-2.0.so.0 [0x70c55]
=========     Host Frame:/lib/x86_64-linux-gnu/libpthread.so.0 [0x76ba]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (clone + 0x6d) [0x10741d]
========= Program hit cudaErrorLaunchFailure (error 4) due to "unspecified launch failure" on CUDA API call to cudaStreamSynchronize. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x359283]
=========     Host Frame:/usr/local/cuda-10.0/lib64/libcudart.so.10.0 (cudaStreamSynchronize + 0x15e) [0x4e5ee]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libnvdsgst_multistream.so [0x13e19]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libnvdsgst_multistream.so [0x14405]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0xa0f31]
=========     Host Frame:/lib/x86_64-linux-gnu/libglib-2.0.so.0 [0x715ee]
=========     Host Frame:/lib/x86_64-linux-gnu/libglib-2.0.so.0 [0x70c55]
=========     Host Frame:/lib/x86_64-linux-gnu/libpthread.so.0 [0x76ba]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (clone + 0x6d) [0x10741d]

cuda-memcheck.log (2.02 MB)

If disable “[vehcio]”, can it also be reproduced ?

yes, that log file is with the plugin disabled

# Copyright (c) 2018 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1920
height=1080
gpu-id=0
# 0 - cuda pinned/host memory
# 1 - cuda device memory
# 2 - cuda unified memory
cuda-memory-type=1

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
#uri=file:///root/DeepStream_Release/samples/streams/sample_720p.mp4
#uri=file://../../brisbane.avi
#uri=rtsp://root:DigExp2907@192.168.1.11/axis-media/media.amp
#uri=file:///root/videos/2019-05-01/11/30_04_2019 11_59_59 PM (UTC+10_00).mkv
uri=file:///root/sample-videos/lane-1-front-20190505.mkv
num-sources=1
gpu-id=0
# 0 - cuda pinned/host memory
# 1 - cuda device memory
# 2 - cuda unified memory
cuda-memory-type=1

[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=1
source-id=0
gpu-id=0
cuda-memory-type=1

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265 3=mpeg4
## only SW mpeg4 is supported right now.
codec=3
sync=0
bitrate=2000000
output-file=out.mp4
source-id=0

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=mp4 2=mkv
container=2
#1=h264 2=h265 3=mpeg4
## only mpeg4 is supported right now.
codec=1
sync=0
bitrate=4000
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5405

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
cuda-memory-type=1

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
#batch-size=4
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
cuda-memory-type=1

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=/root/DeepStream_Release/samples/models/Primary_Detector/resnet10.caffemodel_b4_int8.engine
labelfile-path=/root/DeepStream_Release/samples/models/Primary_Detector/labels.txt
batch-size=4
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=2
gie-unique-id=1
cuda-memory-type=1
config-file=/root/DeepStream_Release/samples/configs/deepstream-app/config_infer_primary.txt

[tracker]
enable=1
tracker-width=640
tracker-height=368
gpu-id=0

[secondary-gie0]
enable=0
model-engine-file=/root/DeepStream_Release/samples/models/Secondary_VehicleTypes/resnet18.caffemodel_b16_int8.engine
labelfile-path=/root/DeepStream_Release/samples/models/Secondary_VehicleTypes/labels.txt
gpu-id=0
batch-size=16
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=/root/DeepStream_Release/samples/configs/deepstream-app/config_infer_secondary_vehicletypes.txt

[secondary-gie1]
enable=0
model-engine-file=/root/DeepStream_Release/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_int8.engine
labelfile-path=/root/DeepStream_Release/samples/models/Secondary_CarColor/labels.txt
batch-size=16
gpu-id=0
gie-unique-id=5
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=/root/DeepStream_Release/samples/configs/deepstream-app/config_infer_secondary_carcolor.txt

[secondary-gie2]
enable=0
model-engine-file=/root/DeepStream_Release/samples/models/Secondary_CarMake/resnet18.caffemodel_b16_int8.engine
labelfile-path=/root/DeepStream_Release/samples/models/Secondary_CarMake/labels.txt
batch-size=16
gpu-id=0
gie-unique-id=6
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=/root/DeepStream_Release/samples/configs/deepstream-app/config_infer_secondary_carmake.txt

[tests]
file-loop=1

[vehcio]
enable=0
processing-width=640
processing-height=480
full-frame=0
unique-id=3
gpu-id=0
config-url=http://172.17.0.2/api/v1/3/shapes/
redis-host=172.17.0.2

Richard
I am afraid we will not fix your problem even if we can reproduce it on Deepstream 3.0.
Our deepstream 4.0 EA release can be available now. Can you try your change on Deepstream 4.0 ? If you still find problem, we will fix it soon.

Yes i can do that.

please let me know how to get access to the DS 4.0 EA

i have signed up for the early access program, awaiting approval

Thanks.

How to join in to the deepsream sdk early access program?