Smart Record leaks file descriptors

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version v5.0 GA
• JetPack Version (valid for Jetson only) v4.4
• TensorRT Version v7.1.3.0
• Issue Type( questions, new requirements, bugs) BUG

Using the attached config file smart_record_file_desciptor_leak.txt (6.0 KB) for 8 rtsp sources with smart-record enabled on interval for each source (to reproduce faster), causes file descriptor leaks on the Jetson Nano using the sample deepstream-app.

After only a few minutes the application crashes, with different errors. Some logs for different runs attached as examples

Note, from my investigation it seems to be leaking 2 file descriptors of type type=STREAM per recording (as can be seen from the command below). This occurs even with a single RTSP stream.

This is critical to me as my application crashes after a certain amount of recordings.

To reproduce, install lsof to check the file descriptors:

sudo apt-get update
sudo apt-get install lsof

After running the deepstream-app using the attached config file:

  • Get the process ID using ps aux | grep deepstream-app
  • Export above process ID as PID, for example export PID=1234
  • Run the following command to watch the file descriptors leaking:
    watch -n 5 "date '+%F %T' && lsof -p $PID | wc -l && lsof -p $PID | grep type=STREAM | wc -l"
2 Likes

@jasonpgf2a have you by any chance noticed this or can you reproduce this. Only asking as you have many posts related to smart-record.

@prominence_ai, this is the memory leak I mentioned.

I haven’t noticed this. My applications are not continually starting and stopping so the issue may be there without me knowing.

I will test using lsof as you suggest and create a lot of detections (which is what triggers smart record for my apps). Will report back next week as I’m on holidays at the moment. ;-)

Thanks @jasonpgf2a, in my applications this is also triggered from detections. Just created a sample config for Nvidia to reproduce using one of their samples.

Enjoy your break :-)

I can reproduce the leak in our board. We will investigate and debug it.

Thanks @Fiona.Chen for confirming and looking into this. Please update once you have made any progress?

Hi @Fiona.Chen, any update on this bug? Has this been logged internally and when can we expect this to be fixed?

Yes, we have internal bug to tracking this problem. Will update here when there is progress.

We have found the root cause. The solution will be released with next release.

That is great news. Thank you for the effort.

By “next release”, do you mean Deepstream 6 or will there be a minor/patch release? Could you provide a timeline on when this will be provided as this is critical to my applications. The only roadmap I could find is Jetson Roadmap | NVIDIA Developer which only mentions Jetpack.

The next release is not DS6, it is 5.0.x serials. The release date is not decided yet, it may be in early Dec.

Thank you, looking forward to the release

Hi @Fiona.Chen ,

Any update on when to expect the release?

The DeepStream 5.0.1 has been published for a period, please check https://developer.nvidia.com/deepstream-getting-started

Hi @kayccc,

Deepstream 5.0.1 has been released on 8 Oct 2020 (Announcing DeepStream 5.0.1 - #3). This issue has only been confirmed by @Fiona.Chen on 27 Oct 2020

I would like know when the next patch release (e.g. 5.0.2) would be available, the release @Fiona.Chen was referring to.