Memory leak in Deepstream 6.0 when running with RTSP Streams

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) : Jetson TX2 NX
• DeepStream Version : Deepstream 6.0
• JetPack Version (valid for Jetson only): 4.6
• Issue Type( questions, new requirements, bugs) : Bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Deepstream python app: deepstream-test3 . Run this app with 4 RTSP streams

When i run the deepstream-test3 python app with 4 RTSP streams, the memory used by the machine is gradully increasing.

Same issue noticed in below links:

NOTE: Upgrading Deepstream into latest version is not possible since the Jetson Tx2 Nx is supported on JP4. 6.3, the corresponding version of DeepStream is 6.0.

3

Could you attach how you monitor the memory and the specific increased data?

4

Hi @yuweiw
For attaching the memory cosumption details , i have to run the app continously for couple of days.

I have observed that my machine memory is keep decreasing when i run df -h command. Available space in my machine was 15GB when the deepstream app starts, after a day the avalable space becomes 14.5GB. (I’m not saving images or anything from this app, using the same deepstream-test3 app)

5

It’s werid that the harddisk memory is keep decreasing. Could you help to check which specific path has experienced memory growth?

6

Hi @yuweiw

Same issue happened when i run in x86 architecture too.

Can you suggest how to check which specific path has experienced memory growth?

7

Basically, you can use the du -sh * to check the size of each file and directory.

8

Here I tried running the same deepstream-test3 app on another aarch64, but getting the same issues.

Memory logs after 2 mins of deepstream-test3 app running:

image
image819×468 56.5 KB

Memory logs after 1 hour of app running:

image
image698×472 55.4 KB

9

Could you try to find the path where hard drive memory is growing step by step? You can find which directory has memory growth, and then enter that directory to continue tracking until the file with memory growth is found.

10

Hi @yuweiw

The memory growth is happened by deepstream docker container. Because app is getting crashed after couple of days, and the memory is not released even after the docker container is stopped, but the memory is released when the docker container is deleted from the machine.

11

OK. Then you can try to find the path where hard drive memory is growing in the docker. Basically, we don’t store anything on the disk for deepstream-test3.
You can also run the DeepStream on your TX2 directly without docker to make a comparison.

12

I’m using the same deepstream-test3 app, i didn’t add any changes to the code. Using the same docker container from NVIDIA NGC.

This issue is happening with RTSP stream. Is it due to memory leak in any elements in the app?

13

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Generally speaking, memory leaks do not lead to the size growth of disk. You can try to run that on your platform directly wihout docker.
We have fixed some memory leak issues in the latest versions, but we cannot provide corresponding patches separately for a particular previous version.

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.