Memory exhaustion when repeating the start/stop cycle of a GStreamer pipeline including nvvideoconvert and any of nvv4l2 encoders

This bug report is similar to the following past issues.

However the memory leak I report here is not included in %MEM observed by the top command and we can observe it just by seeing remaining memory usage with the free command or /proc/meminfo. This would mean the memory leak does not exhaust a general memory heap in a process that is accessed by APIs such as malloc/free but only physical memory is getting exhausted. For this reason, Valgrind cannot observe the memory leak because it just surveys the usage of a process memory heap by monitoring access to the memory heap via general memory management APIs in userspace.

The followings are setup details.

• Hardware Platform (Jetson / GPU) Jetson TX2
• DeepStream Version 5.0.0
• JetPack Version (valid for Jetson only) 4.4

For reproducing the issue, we have to repeat the start/stop cycle of a GStreamer pipeline including the combination of nvvideoconvert and any of nvv4l2 encoders (nvv4l2h264enc or nvv4l2vp8enc or nvv4l2vp9enc). This way is almost the same as this previous issue (Nvvideoconvert element produces memory increases in a create-start-stop-delete sequence). The issue can be reproduced just with the PLAYING/NULL state cycles without recreating a GSteamer pipeline. Additionally I have broken down the way for reproduction to a minimal GStreamer application code.

I have prepared the test code for reproduction in a GitHub repository so that anyone can reproduce the memory leak issue.

Options to select which video encoder to use is added. Also the FullHD resolution is specified as the videotestsrc output because the bigger resolution nvvideoconvert and a nvv4l2 encoder plugin receive, the faster physical memory is getting exhausted, so the issue would be clarified more clearly.

Please build the test code as below.

$ git clone
$ cd nvconv_nvenc_memory_leak
$ make

Then, nvconv_nvenc_loop will be generated as an executable. You should see physical memory is getting exhausted while it is being executed.

How to run the test command for each video encoder as below.

For tesing nvv4l2h264enc

$ ./nvconv_nvenc_loop -c h264

For tesing nvv4l2vp8enc

$ ./nvconv_nvenc_loop -c vp8

For testing nvv4l2vp9enc

$ ./nvconv_nvenc_loop -c vp9

In another terminal, you may want to use the watch command that the free command is passed in order to monitor the memory usage every few seconds.

$ watch free

You should also see OOM killer happens after about 30 minutes.

Could you try to reproduce the issue and investigate how to avoid it?

Can you try this case on the latest DeepStream 6.0.1?

Is that mean you’ve confirmed this memory leak is fixed on that version?

We have fixed some memory leak issue with DeepStream 5.0.

@emi-kkobayashi Can you test on DeepStream 6.0.1? As there were some fixes in DeepStream 5.0, please upgrade DeepStream 6.0.1.

I’ve run you app on DeepStream 6.0.1 for 5 hours, no obvious memory leak.

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.