Is DeepStream4.0 need some optimization?There may be a timestamping problem, or this computer is too slow

tms2003 · May 26, 2020, 2:36pm

I use deepstream4 sample like this:

deepstream-app -c source30_1080p_dec_infer-resnet_tiled_display_int8.txt

and My Computer begin too slowly,I get this console information:

**PERF: FPS 0 (Avg)	FPS 1 (Avg)	FPS 2 (Avg)	FPS 3 (Avg)	FPS 4 (Avg)	FPS 5 (Avg)	FPS 6 (Avg)	FPS 7 (Avg)	FPS 8 (Avg)	FPS 9 (Avg)	FPS 10 (Avg)	FPS 11 (Avg)	FPS 12 (Avg)	FPS 13 (Avg)	FPS 14 (Avg)	FPS 15 (Avg)	FPS 16 (Avg)	FPS 17 (Avg)	FPS 18 (Avg)	FPS 19 (Avg)	FPS 20 (Avg)	FPS 21 (Avg)	FPS 22 (Avg)	FPS 23 (Avg)	FPS 24 (Avg)	FPS 25 (Avg)	FPS 26 (Avg)	FPS 27 (Avg)	FPS 28 (Avg)	FPS 29 (Avg)	
**PERF: 26.95 (26.95)	26.96 (26.96)	26.57 (26.57)	25.65 (25.65)	26.16 (26.16)	25.65 (25.65)	23.45 (23.45)	28.48 (28.48)	25.48 (25.48)	26.54 (26.54)	25.67 (25.67)	28.41 (28.41)	25.27 (25.27)	26.08 (26.08)	25.25 (25.25)	25.98 (25.98)	27.45 (27.45)	24.85 (24.85)	25.29 (25.29)	28.03 (28.03)	25.98 (25.98)	24.19 (24.19)	25.67 (25.67)	26.95 (26.95)	26.08 (26.08)	28.08 (28.08)	26.08 (26.08)	24.85 (24.85)	28.48 (28.48)	27.04 (27.04)	
**PERF: 25.96 (26.82)	26.89 (26.96)	6.91 (15.38)	25.92 (25.68)	13.38 (25.38)	6.60 (14.97)	0.00 (23.45)	0.00 (28.48)	26.05 (25.54)	21.51 (26.17)	5.97 (14.55)	0.00 (28.41)	4.40 (13.48)	26.05 (26.08)	20.08 (24.95)	4.97 (14.08)	0.00 (27.45)	4.04 (13.12)	6.69 (24.15)	0.00 (28.03)	6.28 (14.88)	0.00 (24.19)	25.96 (25.71)	4.14 (13.96)	25.92 (26.06)	0.00 (28.08)	6.60 (15.08)	4.40 (13.38)	0.00 (28.48)	0.00 (27.04)	
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.

I notice it’s run pefect in deepstream3.so,Is DeepStream4.0 need some optimization?

I change sink1 sync=0 and get better.but I notice the avg fps is 13.

My coumpter is ubuntu 18.04 ,with i5 gpu and rtx2060 6g gpu,I notice cpu only 50% and gpu used 45%.

SO ,could i ccelerate the Computing?

amycao · February 1, 2020, 8:42am

Can you change tracker config to this and try again?

[tracker]
enable=1
tracker-width=480
tracker-height=272
ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_nvdcf.so
#ll-lib-file=…/…/…/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
ll-config-file=tracker_config.yml
ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1

amycao · February 1, 2020, 8:45am

Hi
Does this solve your problem? please let us know.

tms2003 · February 1, 2020, 8:45am

@amycao

No,but i restart my computer,and found it’s ok.
source30_1080p_dec_infer-resnet_tiled_display_int8.txt hasn’t tracker ,so ,it ignormal for this.

niuniu_niu1 · February 1, 2020, 9:01am

i also have this problem,so is there any way to solve it?

kayccc · February 4, 2020, 6:51pm

Hi niuniu_niu1

Please try with the latest release - DeepStream SDK 4.0.1. If issue still can be reproduced, please file another topic with clear description.

niuniu_niu1 · February 1, 2020, 9:04am

I have already used DeepStream SDK 4.0.1,but it doesn’t work.so i set batch-size to 25 in file source30_1080p_dec_infer-resnet_tiled_display_int8.txt,then it work.but the FPS can’t reach 30.

mxy · February 1, 2020, 9:08am

I also ran into a similar problem! I am running the latest deepstream 4.0.1 sdk

Everything is fine When I run: deepstream-app -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt

I took the config file, and modified it to display 1x1 instead of 4x4, and set the source to be an rtsp stream coming from an ip camera. I do see frames from the ipcamera, but they are very slow. And I get the errors:
**PERF: FPS 0 (Avg)
**PERF: 93.95 (93.95)
**PERF: 7.46 (8.57)
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.

I’m not sure what’s happening. Could someone please give me some insight?

Below is a snippet of my modifications to the config file.
[i][b][application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://admin:admin@192.168.217.104
num-sources=1
#drop-frame-interval=2
gpu-id=0

(0): memtype_device - Memory type Device

(1): memtype_pinned - Memory type Host Pinned

(2): memtype_unified - Memory type Unified

cudadec-memtype=0[/b]
[/i]

Thanks!

AastaLLL · March 10, 2020, 2:47pm

Hi, niuniu_niu1 and mxy

We try to reproduce this issue in our environment.
Could you share the environment and platform you are using?

Do you use Jetson platform or a desktop environment?
Thanks.

camilosilva91 · May 6, 2020, 6:06pm

I have the same issue. I also get the message “… A lot of buffers are being dropped … There may be a timestamping problem, or this computer is too slow.”

However, I have noticed that only happens when I decode an IP camera using “rtsp”, I have tried several brands, FPS rate, and resolutions (HD,FHD). In the examples “deepstream-test3-app (c++)” and “deepstream-test3 (python3)” the same thing happens (with default configuration files).

But, when I run these same examples with a video file (file: ///…/sample_720p.mp4 [or h264]), it works like a charm, I get 30 fps.

“top” command -> deepstream task: %CPU: 39.1%, %MEM: 8.1%, %Cpu(s): 10.1%
“nvidia-smi” command -> deepstream task: GPU-Util: 22%, GPU-MEM-USAGE: 553MiB

I am using:
deepstream-app version 4.0.2
DeepStreamSDK 4.0.2
TensorRT 6.0.1-1+cuda10.2
Nvidia GTX 1070 (laptop)
CPU i7-7820HK @ 2.90GHz × 8
Ubuntu 18.04

Is there any parameter in these examples to optimize the rtsp (h264) input?

Thanks!

mdegans · March 10, 2020, 3:20pm

I get these same warnings a lot using various network sources, whether rtsp or youtube or whatever. The streams work, maybe dropping a frame here and there, but the spam is more annoying. Is there any way to suppress these warnings (a property on the sink, for example).

amycao · May 6, 2020, 6:06pm

I have the same issue. I also get the message “… A lot of buffers are being dropped … There may be a timestamping problem, or this computer is too slow.”

However, I have noticed that only happens when I decode an IP camera using “rtsp”, I have tried several brands, FPS rate, and resolutions (HD,FHD). In the examples “deepstream-test3-app (c++)” and “deepstream-test3 (python3)” the same thing happens (with default configuration files).

But, when I run these same examples with a video file (file: ///…/sample_720p.mp4 [or h264]), it works like a charm, I get 30 fps.

“top” command -> deepstream task: %CPU: 39.1%, %MEM: 8.1%, %Cpu(s): 10.1%
“nvidia-smi” command -> deepstream task: GPU-Util: 22%, GPU-MEM-USAGE: 553MiB

I am using:
deepstream-app version 4.0.2
DeepStreamSDK 4.0.2
TensorRT 6.0.1-1+cuda10.2
Nvidia GTX 1070 (laptop)
CPU i7-7820HK @ 2.90GHz × 8
Ubuntu 18.04

Is there any parameter in these examples to optimize the rtsp (h264) input?

Thanks!

–> please set ‘sync=false’ to nveglglessink:
g_object_set (G_OBJECT(sink), “sync”, FALSE, NULL);
set streammux ‘live-source=1’
g_object_set (G_OBJECT (streammux), “live-source”, 1, NULL);
and try again.

amycao · March 10, 2020, 3:20pm

In gstreamer frameworks, there is mechanism of doing synchronization according to frame timestamps, it may block and triggers the warning messages, please refer to comment 11#

mdegans · March 10, 2020, 3:20pm

Thanks. I was browsing the documentation on that earlier today. The problem is that if I set sync to false, it breaks playback from files and some other sources like youtube uris (they play as fast as they can).

Is there a way to instruct a sink to drop frames if necessary to resync like maybe with max-lateness? I haven’t experimented with it yet. I should mention i’m using nvmultistreamtiler before my sink. My pipeline is basically [uridecodebin…] ! nvstreammux ! nvmultistreamtiler ! nvoverlaysink

I mention because I am wondering: does nvmultistreamtiler sync in order to join the frames? If so, how and what happens if the timecodes between the streams begin to drift? I want to add raspberry pi sources to test as well and that board lacks a rtc, leading to sync issues with clients. I haven’t tested yet, but I can imagine having some problems.

camilosilva91 · May 6, 2020, 6:06pm

@amycao

Thanks for replying, I forgot to mention that by transmitting “rtsp”, I only got 1 or 2 FPS maximum. But adding “streammux.set_property (‘live-source’, 1)” in the file “deepstream-test3.py”, everything works correctly.

Thanks!!!

amycao · May 6, 2020, 6:06pm

amycao:

I get these same warnings a lot using various network sources, whether rtsp or youtube or whatever. The streams work, maybe dropping a frame here and there, but the spam is more annoying. Is there any way to suppress these warnings (a property on the sink, for example).

In gstreamer frameworks, there is mechanism of doing synchronization according to frame timestamps, it may block and triggers the warning messages, please refer to comment 11#

Thanks. I was browsing the documentation on that earlier today. The problem is that if I set sync to false, it breaks playback from files and some other sources like youtube uris (they play as fast as they can).

Is there a way to instruct a sink to drop frames if necessary to resync like maybe with max-lateness? I haven’t experimented with it yet. I should mention i’m using nvmultistreamtiler before my sink. My pipeline is basically [uridecodebin…] ! nvstreammux ! nvmultistreamtiler ! nvoverlaysink

–> [from amycao] you can refer to this, https://devtalk.nvidia.com/default/topic/1061492/deepstream-sdk/deepstream-sdk-faq/
for dropping frames

I mention because I am wondering: does nvmultistreamtiler sync in order to join the frames? If so, how and what happens if the timecodes between the streams begin to drift? I want to add raspberry pi sources to test as well and that board lacks a rtc, leading to sync issues with clients. I haven’t tested yet, but I can imagine having some problems.

mdegans · March 10, 2020, 3:21pm

Thanks, amycao,

It’s not exactly what I was looking for but I think it can be used in my case as a workaround.

richeyophtc · May 6, 2020, 6:06pm

I encountered the same problem when I run “deepstream-app -c source30_1080p_dec_infer-resnet_tiled_display_int8.txt”.
following the infomation mentioned here， I set “sync=0” and get about 25fps,less than 30fps.
My development environment is：

Intel® Core™ i7-7800X CPU @ 3.50GHz
GeForce RTX 2080Ti x 2
Ubuntu 18.04
DriverL418.87.00
CUDA10.1
tonsorrt6.0.1
cudnn7
deepstream4.0.2
deepstream-app 4.0.2

the GPU-Util is only 18%，Where could the problem be？

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:17:00.0 Off |                  N/A |
| 45%   49C    P2   105W / 250W |   3320MiB / 10989MiB |     18%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:65:00.0  On |                  N/A |
| 40%   42C    P2    70W / 250W |   1554MiB / 10986MiB |      7%      Default |
+-------------------------------+----------------------+----------------------+

when I enable sink2 to RTSP streaming,the fps drops to about 5 fps below and the GPU-Util is only 3%.

|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:17:00.0 Off |                  N/A |
| 47%   49C    P2    57W / 250W |   3333MiB / 10989MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:65:00.0  On |                  N/A |
| 39%   37C    P8    26W / 250W |   1583MiB / 10986MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+                                                                              
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     22006      C   python3                                      631MiB |
|    0     29399      C   deepstream-app                              2564MiB |
|    1      1212      G   /usr/lib/xorg/Xorg                            40MiB |
|    1      1523      G   /usr/bin/gnome-shell                          59MiB |
|    1      2581      G   /usr/lib/xorg/Xorg                           675MiB |
|    1      2732      G   /usr/bin/gnome-shell                         371MiB |
|    1      3163      G   ...AAAAAAAAAAAAAAgAAAAAAAAA --shared-files   294MiB |
|    1     22561      G   ...uest-channel-token=12432704521303380480   127MiB |
+-----------------------------------------------------------------------------+

here is my config file:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=5
columns=6
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../streams/sample_1080p_h264.mp4
num-sources=15
#drop-frame-interval=2
#num-extra-surfaces=15
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../streams/sample_1080p_h264.mp4
num-sources=15
#drop-frame-interval=2
#num-extra-surfaces=15
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
sync=0
#iframeinterval=10
bitrate=2000000
output-file=out.mp4
source-id=0

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
sync=0
bitrate=4000000
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=30
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b30_int8.engine
#Required to display the PGIE labels, should be added even when using config-file
#property
batch-size=30
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
#Required by the app for SGIE, when used along with config-file property
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tests]
file-loop=0

could you give me some suggestions？
Thanks.

mdegans · March 10, 2020, 3:21pm

I haven’t looked at the pipeline of the DeepStream reference app, but in my case I am going to put my network/file sources/sinks in their own threads by using various queue elements before and after my inference Bin.

I am thinking a multiqueue before my stream muxer and queues before my network/file sinks. I don’t know if you can do that with the reference app without modification. I will update if the approach helps with this issue.

Edit: so it seems uridecodebin (or rather decodebin) creates a multiqueue as it is, so it wouldn’t help much. gonna try on the sink.

mchi · April 27, 2020, 1:07am