Please provide complete information as applicable to your setup.
**• Hardware Platform ----------> GPU
**• DeepStream Version ----------> 7.0
• TensorRT Version ------------> 8.6
**• NVIDIA GPU Driver Version -----> 555
Deepstream L4 can support 81 (H265) rtsp :- With pgie + tracker + 2 sgie model.
We trying in L4 deepstream-rtsp-in-out apps running 80 streams But only getting 10-15 FPS and decoder is using less than 50%, Can you suggest me why this thing is happening ?
How I can benchmark and achieve same performance !!!
Are you talking about Performance — DeepStream documentation 6.4 documentation?
What kind of 80 streams(local files, RTSP streams, …？）? With the same models in Performance — DeepStream documentation 6.4 documentation? With the same configurations in Performance — DeepStream documentation 6.4 documentation?
We have provided the details in Performance — DeepStream documentation 6.4 documentation, please use exactly the same sources, models and configurations with deepstream-app sample to get the same benchmark.
What is your purpose of the benchmark? The data in Performance — DeepStream documentation 6.4 documentation is for the whole pipeline, it may be different with different models, different video streams and different pipeline or configurations. What is your use case?
Hi @Fiona.Chen
I was running the same thing.
I am getting 25 FPS for all 81 streams which video you are using as “streams/sample_1080p_h265.mp4”,
for this logs is
exp_benchmarklogs_file.txt (163.7 KB)
When I enable file-loop =1
FPS goes down to 15 FPS
I am attaching the logs for fileloop=1 and " source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt" what I changed
source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt (5.2 KB)
exp_benchmarklogs.txt (207.1 KB)
For better understanding we ran our rtsp streams, there is FPS drop to 15 FPS.
Can you help us now why this behaviour is coming ?
Please change the following parts:
[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink/nv3dsink (Jetson only) 3=File
type=1
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0
[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
buffer-pool-size=4
batch-size=81
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0
exp_benchmarklogs_sync0.txt (898.3 KB)
This experiment with sync0 also same results has not been reproduce of 30FPS
Please see why the FPS is decrease to 14-15FPS and while decoder is only 30%.
Can you please suggest the FPS we are expecting for 81 streams form nvidia benchmark on L4 ?
I am attaching one stream graph how it’s behaving
Can you please run this experiment where file-loop=1
for longer period of time or a longer video, please as you can see the graphs also we are also getting 25FPS at the beginning as the time progress FPS is decreasing
Please share your logs with “file-loop=1” and run it longer period of time.
This is captured after 30 minutes running with “file-loop=1”.
@Fiona.Chen
Our CPU -----> Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
I’m using this CPU and one GPU is L4.
I also ran the same thing for me FPS falling time by time. I really want to understand why it’s coming ! Does CPU impact ? If yes ! Why ? If No How I can improve the FPS ?
Please monitor the CPU loading and GPU loading when you run the “file-loop=1” case to find out more clues.
Hi @Fiona.Chen
I am sharing with you FPS.csv and docker stats.csv and nvidia-smi_dmon.csv
Can you look into it and share me some inside where we can fix this issue !
L4_nvidia.zip (80.0 KB)
The CPU loading looks OK. The GPU is full loaded. No clue to show there is any issue.
@Fiona.Chen
Our driver version is “Driver Version: 555.58” , I show your version is 550,
I understand It might not impact ! Does driver can be a problem ?
I am asking you figure it out what is problem ? same GPU different performance !
We follow deepstream document to install and set up the VM, Did you have something extra ?
to narrow down where is the problem !
It is not verified in our side.
There is no clear clue to identify the root cause. How many L4 GPUs do you have? Can you try on other L4 device?
We have only one L4 ! I don’t have any option. @Fiona.Chen
agu_5_file_loop_0.txt (163.7 KB)
If I give file_loop = 0
and run that is the out.txt here also FPS decreasing !
any env level changes ?
any GLOBAL_FLAG NEEDS TO BE ENABLED ??
Are you using Deepstream 6.4 or Deepstream 7.0 ?
