This bug report is similar to the following past issues.
- Nvvideoconvert element produces memory increases in a create-start-stop-delete sequence
- Memory Leak in Video Recording
However the memory leak I report here is not included in %MEM observed by the top command and we can observe it just by seeing remaining memory usage with the free command or /proc/meminfo. This would mean the memory leak does not exhaust a general memory heap in a process that is accessed by APIs such as malloc/free but only physical memory is getting exhausted. For this reason, Valgrind cannot observe the memory leak because it just surveys the usage of a process memory heap by monitoring access to the memory heap via general memory management APIs in userspace.
The followings are setup details.
• Hardware Platform (Jetson / GPU) Jetson TX2
• DeepStream Version 5.0.0
• JetPack Version (valid for Jetson only) 4.4
For reproducing the issue, we have to repeat the start/stop cycle of a GStreamer pipeline including the combination of nvvideoconvert and any of nvv4l2 encoders (nvv4l2h264enc or nvv4l2vp8enc or nvv4l2vp9enc). This way is almost the same as this previous issue (Nvvideoconvert element produces memory increases in a create-start-stop-delete sequence). The issue can be reproduced just with the PLAYING/NULL state cycles without recreating a GSteamer pipeline. Additionally I have broken down the way for reproduction to a minimal GStreamer application code.
I have prepared the test code for reproduction in a GitHub repository so that anyone can reproduce the memory leak issue.
Options to select which video encoder to use is added. Also the FullHD resolution is specified as the videotestsrc output because the bigger resolution nvvideoconvert and a nvv4l2 encoder plugin receive, the faster physical memory is getting exhausted, so the issue would be clarified more clearly.
Please build the test code as below.
$ git clone https://github.com/emi-kkobayashi/nvconv_nvenc_memory_leak
$ cd nvconv_nvenc_memory_leak
$ make
Then, nvconv_nvenc_loop will be generated as an executable. You should see physical memory is getting exhausted while it is being executed.
How to run the test command for each video encoder as below.
For tesing nvv4l2h264enc
$ ./nvconv_nvenc_loop -c h264
For tesing nvv4l2vp8enc
$ ./nvconv_nvenc_loop -c vp8
For testing nvv4l2vp9enc
$ ./nvconv_nvenc_loop -c vp9
In another terminal, you may want to use the watch command that the free command is passed in order to monitor the memory usage every few seconds.
$ watch free
You should also see OOM killer happens after about 30 minutes.
Could you try to reproduce the issue and investigate how to avoid it?