0 FPS after several hours of running fine

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson Nano
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) 32.4

Sometimes when I run deepstream-app for a long period of time it will suddenly and unexpectedly drop to 0 fps. I have not been able to reliably reproduce the error, it happens randomly.

Any tips or suggestions on how to troubleshoot this would be greatly appreciated.

Edit: To clarify, the app continues to run, but just continuously shows 0fps and never recovers. It requires restarting the app to resolve the issue.

Edit: I’m using a basic pipeline with a single CSI camera source, and RTSP, file, and msgbroker sinks, as well as inference element.

1 Like
3

the popular reason is streams send EOS from server or disconnection. you need to write your application to handle that, reconnect to stream which had stopped.

4

Do you mean I can narrow down the issue to my source element? Or is is also possible a sink element can be causing an issue?

The only source I use is the CSI camera. Do you know why it would send EOS after running fine for a long period of time?

5

Hi,
Can you get the tegrastats log when the issue happened?
sudo tegrastats

6

@Amycao I cannot reliably repoduce the error, but I will try to get the tegrastats logs from when this happens.

I’m using a basic pipeline with a single CSI camera source, and RTSP, file, and msgbroker sinks, as well as inference element.

Do you have any other suggestions on how to troubleshoot this?

1 Like
7

Hi @Amycao I work with @mattcarp88 on this project.

Just had this issue happen again, for whatever reason it’s taking about 12-24 hours for the issue to happen, but it’s not predictable and sometimes it won’t happen for several days. Even if we ask tegrastats to output to a log file, tegrastats does not provide a timestamp with each logged entry, so it would be hard to find the needle in the haystack since the failure is not reproducible and doesn’t occur for several hours or days. Is there a specific metric in tegrastats that you think would give us a clue here?

Or, as @mattcarp88 asked, do you have any other suggestions for how we can further debug this? We’ve enabled additional debug logging out of DeepStream and have been able to find the time when the FPS went from 20FPS to 0FPS, but there were no warning or error messages reported at that time.

Thanks

1 Like
8

@harryhsl8c @mattcarp88
The tegrastats log can tell whether or not if the rtsp source have output at the time of FPS went down to 0, DEC engine should be 0 at that time.

9

@Amycao We are not using RTSP source, only a single CSI camera source.

10

Oh, ok, sorry.
first you should determine which module cause the issue. that’s why i asked about tegrastats, which may give some clues when the issue happened.
or you can change the module in your application one by one to find the module which cause the issue.

12

@Amycao Thank you for your help on this. Can you suggest how we can troubleshoot this issue?

13

Did you try the suggestion from comment 9#？

14

Yes, I would like to try your suggestion from #9. How would you suggest I find the module causing the issue?

Again, I am using the deepstream reference app.

15

I am not sure, but suggest you try with source first.