Deepstream FPS drops when i add more and more RTSP streams

Have you set up a suitable batch-size for nvstreammux? You can refer to the link below first: https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/34

I have set the batch size as follows.

[streammux]
batch-size = 8

[primary-gie]
batch-size = 8

Here i have set batch size as 8 because I’m working with 8 streams.

Yes. But you add more streams than 8. You also can try to set the batched-push-timeout parameters.

All the RTSP streams or feeds with which I’m working are set to 25 FPS and the calculation for batched-push-timeout would be 10,00,000 / 25 = 40,000.

As i increase the streams I’m making sure that batch-size for streammux and primary-gie is same as the number of streams.

FPS in the app drops as i increase the streams. The same is mentioned above.

OK. Could you attach your piepline by referring the https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/10.
You can also observe the load situation of your device. Perhaps the device is overloaded, causing slow processing speed.

OK. I’ll check and update.

Do you still need support for this topic? Or should we close it? Thanks.

I still need the support for this topic.

Overloading may be an issue so I disabled all the detection models and still observed the same results.

So after disabling all the detection models, do you observe the load situation of your device again? Could you attach your deepstream-app config file?
And what is the specific model of your graphics card?

I’m using the default deepstream app and deepstream test 5 app for this testing.

In the config file i have disabled [message-converter] and [message-consumer0]

I have made sure that batch size of streammux is same as that of the number of sources and live-source=, batched-push-timeout=40000 as i am working with 25 FPS RTSP feed.

I have A5000 GPU.

I Observed the following in some of the tests which i have done.

I created multiple config files where i limited one config file with only 4 streams because in the case scenario observed above i was getting 24 FPS for 6 streams.

I created two config files having 4 streams each and i got an average FPS of 20 to 21 FPS for both the configs when i run simultaneously.

similarly when i created 3 config files of 4 streams each i am getting average FPS of 16 to 17 FPS.

similarly when i created 4 config files of 4 streams each i am getting average FPS of 15 to 16 FPS.

But i observed that even the FPS is high I am getting glitch in the deepstream app. I did verified the stream side by side in VLC media player as well. There appears to be no glitch in the VLC media player.

When i created one single config i.e for all 16 streams the average FPS i got was 8 to 9.

I am using nvidia-smi to verify the GPU RAM consumption and usage.

Is this the correct way to identify the GPU load situation ?

Could you attach your config file to us? What is the resolution and bitrate of your camera video?
You can use the nvidia-smi dmon to identify the GPU load situation and the htop to identify the CPU load situation.

Could you attach your config file to us.
→ It is the same config file which is provided with the deepstream.
config.txt (3.5 KB)

What is the resolution and bitrate of your camera video.
→ Resolution: 1920x1080
bitrate: 4096 Kbps.

Output of nvidia-smi dmon while running on 12 streams.
Screenshot from 2023-11-01 17-08-51

OK. Could you also check the CPU loading with htop?
Could you change the type of sink0 to Fakesink and if the fps changes with the number of sources increasing?
You can also open the latency log of the deepstream-app by referring to Enable Latency measurement for deepstream sample apps.

Below are the images of htop.

  1. Before restart of the server.

  2. After restart of the server.

When sink0 is set to Fakesink the FPS improvised by 1 to 2 FPS. But i want the tiled display to be enabled.

latency is ranging from 150 to 200 ( ms ).

What did you do before restart of the server and after restart of the server? Just from the first image, your cpu load average exceeds 90%. The CPU may be your performance bottleneck.

And you can check which plugin has a higher latency.

Before and after restart of the server i have done nothing. Same application works.

I have created 3 configs where 1 config has 4 streams i.e total 12 cameras and i am getting avg FPS of 16 to 17 but, I’m observing glitch as well. CPU load is 70 % and nvidia-smi dmon output is similar as above.

These are the issues which i am facing.

  1. There is drop in FPS and also high glitch in the output feed when all the streams are in same / single config file.
  2. I do get some improvements in the FPS when i split them but the glitch still remains.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

I have tried that in my environment, there are no fps drop issue. Could you try that in my way?

  • refer to the Build rtsp server to build rtsp server and generate 30 rtsp streams
  • use the source30_1080p_dec_infer-resnet_tiled_display_int8.txt file in the samples/configs/deepstream-app dir, modify that with the 30 rtsp sources
  • deepstream-app -c source30_1080p_dec_infer-resnet_tiled_display_int8.txt

source30_1080p_dec_infer-resnet_tiled_display_int8.txt (13.5 KB)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.