Deepstream-app buffer caching observed when using YoloV3 with multiple RTSP output streams

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (GPU RTX2080 Ti)
• DeepStream Version 5.0 DP
• TensorRT Version 7.0
• Driver Version: 450.36.06 CUDA Version: 11.0

When YoloV3 is used in deepstream-app along with objectDetector_Yolo parser plugin provided as sample, the output has jitter where the output appears to be rewinding backwards for a fraction of a second and then proceeds. It appears as though some buffers in some cache are being re-processed. The deepstream-app configuration has 4 RTSP input streams (coming from a sinlg eVLC server) and 4 RTSP output streams, with primary inference done with batch size of 4 and INT8 processing.

Has anyone faced this issue? This issue in not observed when resnet10 model (provided under samples folder ) is used.

What would be going wrong?

Part of the config file is below:

[sink3]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=1
sync=0
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0

set below properties in case of RTSPStreaming

rtsp-port=5003
udp-port=5403
source-id=3
output-file=out.mp4
container=1

[osd]
enable=1
gpu-id=0
border-width=1
text-size=8
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=1
clock-x-offset=00
clock-y-offset=00
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000

Set muxer output width and height

width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

If set to TRUE, system timestamp will be attached as ntp timestamp

If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached

attach-sys-ts-as-ntp=1

#YoloV3
[primary-gie]
enable=1
gpu-id=0
model-engine-file=model_b4_gpu0_int8.engine
labelfile-path=labels.txt
batch-size=4
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=1
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_yoloV3.txt

[tracker]
enable=1

For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively

tracker-width=640
tracker-height=384
#ll-lib-file=…/lib/libnvds_mot_iou.so
#ll-lib-file=…/lib/libnvds_nvdcf.so
ll-lib-file=…/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
#ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1

3

Hi @deepak,
could you change “sync=0” to “sync=1” in [sink3] and check if it’s still reproduciable?

4

Hi,

Setting synch=1 for RTSP sink3 seems to solve the issue of buffer caching (repeated processing). Could you please explain the reason behind setting it to 1?

However, another problem cropped up: We use 4 RTSP streams (with same H264, 1080p content from the sample streams) from an RTSP Server (VLC) as inputs (source0 to source3.)We notice that there is significant delay (upto 3 seconds) in sink3 RTSP output as compared to other RTSP streams. This is much less in case of other RTSP streams which use hardware encoder (RTX 2080 Ti which limits HW encoding to 3)

How to avoid this delay in the output stream?

Thanks for your inputs.

5

Hi @deepak,
Regarding the delay, could you refer to below FAQ to narrow down the latency ?

Thanks!

https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_troubleshooting.html

The DeepStream application is running slowly.

•Solution1: One of the plugins in the pipeline may be running slowly.

You can measure the latency of each plugin in the pipeline to determine whether one of them is slow.

•To enable frame latency measurement, run this command on the console:

$ export NVDS_ENABLE_LATENCY_MEASUREMENT=1

•To enable latency for all plugins, run this command on the console:

$ export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1

6

Hi!

Thanks again!
I will look into this and measure the latency of each plugin. Currently only streammux, nvinfer, tracker and OSD are included.

Also, we noticed another aspect: If we increase the number of sources to 8 with batch size of 8 and with 6 sinks, the buffer caching re-occurs (and intermittently, few frames are re-processed and the output looks like it has been rewind few frames backwards and repeated the processing) . Reducing the number of sources to 6 does not help either. The CPU is about 40% and GPU is about 70-75%. In other words, though the GPU and CPU have the capacity, they are not being utilized. This doesn’t seem right. The symptoms loo like some kind of buffering rather than processing delay/latency. Is that correct?

7

could you share the config file?

Thanks!

8

Hi,

Just to give a quick background, we are trying to assess the performance levels of NVIDIA GPUs for an analytics application. The Resnet10 seems to be too simplified model for any practical use. Similarly, using a tiled output is of no use in many scenarios. The sample deepstream-app performs reasonably well (and appears to process even upto 30 FPS). What we want to do is process 8+ independent RTSP streams and output 8+ RTSP streams.

We have modified the application slightly:
(1) To fix the issue of application crash when more than one sink is used in the code create_pipeline() of deepstream_app.c, patch released by NVIDIA is used.
(2) We also have included the NVIDIA nvdsanalytics and nvof plugins in the pipeline. However, they are disabled as far this testing (as seen in the config file).

The machine configuration is:
CPU: Ryzen 9 3900X, (12 Cores, 24 Threads)
Memory: 64GB
GPU: ZOTAC GAMING GeForce RTX 2080 Ti Twin Fan 11GB GDDR6

With this configuration, when we run the application, we notice:
CPU: about 30% with load average between 4-8, GPU: about 50%, just 30% memory being used and FPS reduced to about 15. The problem is not just the low FPS but also the Jitter, large delay in output streams when compared to inputs. So, something is not right with the model/pipeline or we are missing something. Appreciate your inputs. Here is the config file:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=0
rows=2
columns=2
width=1280
height=720
gpu-id=0
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/streams/sample_1080p_h264.mp4
#uri=file://…/streams/Dog4727.mp4
uri=rtsp://localhost:9000/
#uri=rtsp://192.168.2.68:9000/
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/streams/sample_1080p_h264.mp4
#uri=file://…/streams/Dog4727.mp4
uri=rtsp://localhost:9000/
#uri=rtsp://192.168.2.68:9000/
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0

[source2]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/streams/sample_1080p_h264.mp4
#uri=file://…/streams/Dog4727.mp4
uri=rtsp://localhost:9000/
#uri=rtsp://192.168.2.68:9000/
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0

[source3]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/streams/sample_1080p_h264.mp4
#uri=file://…/streams/Dog4727.mp4
uri=rtsp://localhost:9000/
#uri=rtsp://192.168.2.68:9000/
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0

[source4]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/streams/sample_1080p_h264.mp4
#uri=file://…/streams/Dog4727.mp4
uri=rtsp://localhost:9000/
#uri=rtsp://192.168.2.68:9000/
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0
#latency=1000

[source5]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/streams/sample_1080p_h264.mp4
#uri=file://…/streams/Dog4727.mp4
uri=rtsp://localhost:9000/
#uri=rtsp://192.168.2.68:9000/
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0
#latency=1000

[source6]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/streams/sample_1080p_h264.mp4
#uri=file://…/streams/Dog4727.mp4
uri=rtsp://localhost:9000/
#uri=rtsp://192.168.2.68:9000/
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0
#latency=1000

[source7]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/streams/sample_1080p_h264.mp4
#uri=file://…/streams/Dog4727.mp4
uri=rtsp://localhost:9000/
#uri=rtsp://192.168.2.68:9000/
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0
#latency=1000

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5000
udp-port=5400
source-id=0
#link-to-demux=1

[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5001
udp-port=5401
source-id=1
#link-to-demux=1

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5002
udp-port=5402
source-id=2

[sink3]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=1
sync=1
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5003
udp-port=5403
source-id=3
output-file=out.mp4
container=1

[sink4]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=1
sync=1
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5004
udp-port=5404
source-id=4
output-file=out.mp4
container=1

[sink5]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=1
sync=1
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5005
udp-port=5405
source-id=5
output-file=out.mp4
container=1

[sink6]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=1
sync=1
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5006
udp-port=5406
source-id=6
output-file=out.mp4
container=1

[sink7]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=1
sync=1
#iframeinterval=10
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5007
udp-port=5407
source-id=6
output-file=out.mp4
container=1

[osd]
enable=1
gpu-id=0
border-width=1
text-size=8
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=1
clock-x-offset=00
clock-y-offset=00
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=8
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
##Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
##If set to TRUE, system timestamp will be attached as ntp timestamp
##If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
#attach-sys-ts-as-ntp=1

#YoloV3
[primary-gie]
enable=1
gpu-id=0
model-engine-file=model_b8_gpu0_int8.engine
labelfile-path=labels.txt
batch-size=8
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_yoloV3.txt

[tracker]
enable=1
#For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
#ll-lib-file=…/lib/libnvds_mot_iou.so
#ll-lib-file=…/lib/libnvds_nvdcf.so
ll-lib-file=…/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
#ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1

[secondary-gie0]
enable=0
model-engine-file=…/models/Secondary_VehicleTypes/resnet18.caffemodel_b16_gpu0_int8.engine
gpu-id=0
batch-size=16
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=2;
config-file=config_infer_secondary_vehicletypes.txt

[secondary-gie1]
enable=0
model-engine-file=…/models/Secondary_CarColor/resnet18.caffemodel_b16_gpu0_int8.engine
batch-size=16
gpu-id=0
gie-unique-id=5
operate-on-gie-id=1
operate-on-class-ids=2;
config-file=config_infer_secondary_carcolor.txt

[secondary-gie2]
enable=0
model-engine-file=…/models/Secondary_CarMake/resnet18.caffemodel_b16_gpu0_int8.engine
batch-size=16
gpu-id=0
gie-unique-id=6
operate-on-gie-id=1
operate-on-class-ids=2;
config-file=config_infer_secondary_carmake.txt

[secondary-gie3]
enable=0
gpu-id=0
#model-engine-file=
batch-size=16
#Required by the app for OSD, not a plugin property
bbox-border-color0=0;0;1;1
bbox-border-color1=0;0;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;0;1;1
interval=0
gie-unique-id=8
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=face_detection.txt

[tests]
file-loop=1

[analytics]
enable=0
config-file=…/configs/config_nvdsanalytics.txt

[opticalflow]
enable=0
motion-detection-threshold=50
motion-block-threshold=0

9

BTW, one correction in the above file: source6, and source7 RTSP streams are also enabled, to match the batch-size. All output types are RTSP. Performance level remain the same.

We stream the sample_1080p_h264.mp4 from VLC Player on the same machine. The sources are configured to connect to the RTSP server. The outputs from the pipeline are viewed on the same machine in another instance of VLC Player (on the same machine or on another machine). While the input stream is verified to be smooth, the outputs are jittery and with huge delay as explained earlier.

When all SW ENcoders are used, the FPS throughput for 8 streams drops to 10. (with about 40% GPU and 50% CPU being utilized by deepstream-app with load average of 12-15), which is strange.

The latency measured indicate (using 3 HW and 3 SW Encoders) that the frame latencies are of the order of 500-600ms. Even when all SW encoders are used, with so much capacity available with GPU and CPU and memory, the output streams should only be delayed but should not be Jittery or re-processed/buffered.

YoloV3 model and code is used from the objectDetector_Yolo sample.

10

Hi,

I was hoping to get some response by now.

This issue occurs on Jetson Nano as well when we have more than 1 source and that too with resnet10 !
Here is the configuration:

Input: 1,4,8 RTSP Streams (1080p, 30FPS).
Output: 1,4,8 RTSP Streams
Pipeline: deepstream-app configuration file with: Mux->Primary Detector (resnet10)->Tracker->(1,4,8) RTSP Sinks (NO Tiler, NO Secondary Inferences)

Each of the sink correspond to respective source, sync=1 for each of the sink.

The issue is not seen when there is only one RTSP source and one RTSP sink.

The output stream when observed from another PC on the same LAN has huge distortions, garbled, streaks ,and buffer-reprocessing described earlier.

When we have 4 Source and 4 sinks are enabled with batch-size=4, the problem occurs.
CPU: 50-55%, Memory: 55%, load Average: 5-6, FPS Throughput: ~30

When we have 8 Sources and 1 sink is enabled with batch-size=8, this doesn’t occur.
CPU ~55-60%, Memory ~50%, Load Average: ~7 and FPS Throughput: ~30

When we have 8 Sources and 2 sinks are enabled with batch-size=8, the problem occurs.
CPU: 60%, Memory 55%, Load Average ~8, FPS reduced to 27

the issue won’t go away if sync=0 or 1.

So, clearly, the issue is occurring when more than one RTSP sink is enabled on different hardware platforms and even with resnet10, indicating an issue with the NVIDIA plugins OR the way the pipeline is configured due to limited knowledge.

Could some one from NVIDIA please respond ?

11

Hi @deepak
Sorry for delay! We are trying to find two RTSP source to repo this issue.

Per your pipeline: Mux->Primary Detector (resnet10)->Tracker->(1,4,8) RTSP Sinks (NO Tiler, NO Secondary Inferences)

Can we see this issue if replacing “(1,4,8) RTSP Sinks” with “local display”, e.g. HDMI ?

Thanks!

12

Hi,

Thanks for the response.

The issue is easily reproducible with one VLC acting as a streaming server and provide the sources as below:

cvlc sample_1080p_h264.mp4 :sout=#gather:transcode{scodec=none}:rtp{sdp=rtsp://:9000/} :no-sout-all :sout-keep :loop
(You can use a longer duration MP4 file if necessary)

The above source can be used as input (using the uri shown in the config file in earlier message).

Another VLC (either on the same machine, if it has a sufficient power or on another PC) can be used to view the output (and input and you will see a huge difference in the quality, delays, caching).

We have also observed the issue when running the above pipeline on a PC (as reported in the first message) with VLC on the same machine. In other words, the network delays were (almost) not there.

BTW, on Jetson, we tried again using resnet10, batch-size=4, live-source=1, sync=1 for all sinks.
This did not have any impact and the output remains poor.

13

Hi @mchi
Just in case you may have not noticed: I have updated the information in my previous reply regarding the batch-size, sync etc.

14

Hi @mchi

This issue also occurs when we use H264, 720p , 30FPS, RTSP source

15

Hi @deepak,
is it possible just share me a repo packge for the repo on NANO?

Thanks!

16

Hi @mchi

Which files do you need? Would the sources and config files suffice? how do I upload to NVIDIA?

17

I got DeepStream 5.0 + NANO setup.
I hope to know which fileand the command I need to reproduce the issue.
You can find the “Upload” button on the top of the comment box. If it’s not convinent to share here, you could send me with private message and attach the files.

Could you help?

Thanks!

18

Hi @mchi

You can use the deepstream-app -c

A sample config file for nano is below (you need to change the IP addresses as per your setup)

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=0
rows=4
columns=2
width=1280
height=720
gpu-id=0

nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/…/streams/sample_1080p_h264.mp4
uri=rtsp://192.168.1.7:9000/test
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/…/streams/sample_1080p_h264.mp4
uri=rtsp://192.168.1.7:9000/test
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0

[source2]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/…/streams/sample_1080p_h264.mp4
uri=rtsp://192.168.1.7:9000/test
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0

[source3]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://…/…/streams/sample_1080p_h264.mp4
uri=rtsp://192.168.1.7:9000/test
num-sources=1
#drop-frame-interval=2
gpu-id=0
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5000
udp-port=5400
source-id=0

[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5001
udp-port=5401
source-id=1

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5002
udp-port=5402
source-id=2

[sink3]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
#set below properties in case of RTSPStreaming
rtsp-port=5003
udp-port=5403
source-id=3

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
##Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
##If set to TRUE, system timestamp will be attached as ntp timestamp
##If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
attach-sys-ts-as-ntp=1

#config-file property is mandatory for any gie section.
#Other properties are optional and if set will override the properties set in
#the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=…/…/models/Primary_Detector_Nano/resnet10.caffemodel_b4_gpu0_fp16.engine
batch-size=4
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=4
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_nano.txt

[tracker]
enable=1
#For the case of NvDCF tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=480
tracker-height=288
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_iou.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
#ll-config-file required for IOU only
#ll-config-file=iou_config.txt
gpu-id=0

[tests]
file-loop=1

19

there are several sinks, can you see the issue on any sink?

I tried local display to view one stream, it works well. no caching observed.

20

You can use any RTSP sink (there should be more then one RTSP sink). You will observe the issue

21

Hi @mchi

FYI, the problem does NOT occur if multiple instances of deepstream-app is executed with each instance with ONE 1080p x 30fps RTSP input stream and ONE RTSP output stream. The Pipeline is: RTSP input->Mux->Infer->Track->OSD->Demux->RTSP output pipeline. This may provide some additional information/confirmation that the issue is either with the buffering or plugins or the config file or our understanding of the framework.

Any speedy resolution to this issue and help would be highly appreciated as we are trying to evaluate suitability of NVIDIA hardware/software for the Video Analytics application deployment.

This is the biggest bottleneck we have in this regard.