Possible GPU memory leak after unreferencing Pipeline

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.3
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
535.54.03
• Issue Type( questions, new requirements, bugs)
Possible GPU memory
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I’m reporting this for another company using our open source, so I apologize for not having all the information.

They have an application that creates and destroys DeepStream Pipelines on demand or on event. They’re reporting that the application seems to be leaking GPU memory overtime.

They believe they’ve recreated the issue using the deepstream-test-1 example, by iterating over the main function consecutive times. Here’s the updated the cpp file (renamed for upload).
deepstream_test1_app.c.txt (12.0 KB)

I believe they’re using the SMI tool to measure usage.

Let me know if this test is invalid in anyway, or if there is anything else you need.

Thanks,
Rober3t.

3

Thanks for the sharing! what is SMI tool? how did you start the application? are you using nvinfer or nvinferserver?

  1. could help to narrow down this issue? for example, simplifying the code, using fakesink or “nvinfer + fakesink” to find which NV plugin has memory leak.
  2. please refer to this topic. can you provide more logs by nvmemstat ? it can save the memory usage at intervals. can you provide more valgrind logs by this method vagrid ? it can give memory leak details.
4

hi，it’s me. i just modify the example deepstream-test1 like this

image
image971×515 10.9 KB

5

Then run the command: ./deepstream-test1-app …/…/…/…/samples/streams/sample_720p.h264

6

Then watching the usage of gpu memory time and time again like this :

image
image981×372 10.4 KB

7

They are using the default

  NvDsGieType pgie_type = NVDS_GIE_PLUGIN_INFER;

See additional comments by @446073615

8

after replacing nveglglessink with fakesink on DS6.4, I did not observe obvious memory leak. here is the log log-2.23.txt (22.6 KB).

9

@fanzh are you saying that you only saw the memory leak after switching to fakesink, or you saw the leak before and after?

And is this an nvbug then?

10

sorry. I tested deepstream_test1_app.c.txt on dgpu with DS6.4. the only new modification is replacing nveglglessink with fakesink(I did not test nveglglessink ). then I did not observe obvious memory leak. in my log log-2.23.txt, the memory returned 188MB after every loop.

12

After comprehensive testing, I have finally determined that this issue no longer exists in Deepstream 6.4. Great work,thanks!

13

Hi,sir ,you said you did not observe obvious memory leak with deepstream6.4. It is all right with gpu memory. But there must be something wrong with the increasing cpu memory. I find that the cpu memory will not be released completely when endding the loop every time

image
image1231×701 68.3 KB

14

Thanks for the sharing! let me have a try.

15
  1. I tested deepstream_test1_app.c.txt on dgpu with DS6.4. the only new modification is replacing nveglglessink with fakesink(I did not test nveglglessink ). then I did not observe obvious CPU memory leak using top command-line. in my log memory-0305.txt (6.7 KB), the RES returned 2.2g after every loop.
  2. could you share your use scenario? why do you need to restart pipeline many times without ending the application? I also tested with htop. after 240 loops, the Mem usage as you highlighted in the screenshot increased from 4.47G to 4.52G.
16

Thanks ,sir. My application will deploy multiple scenes on a NVIDIA graphics card, each scene being a pipeline. Sometimes I need to release some pipelines and then start another pipeline, I don’t want to end the entire program when releasing one pipeline because the other pipelines are still running

17

thanks for the sharing! we usually use valgrind mentioned above to check memory leak, from the 20 loops log 20.txt (22.4 KB). there is one memory leak point in DeepStream6.4. we will fix it in the latter versions.

18

Thanks . By the way, may i ask when the new version will be released?

19

the release date has not yet been determined. BTW, the memory leak I mentioned is not to much. only 1k bytes in 20 loops.

21 
==15768==    by 0x4A6C1E9: g_list_prepend (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7200.4)
==15768==    by 0xCDC339C: ??? (in /opt/nvidia/deepstream/deepstream-6.4/lib/gst-plugins/libnvdsgst_multistream.so)

We have fixed this memory leak issue internally. Please wait for the latter release.

22

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.