Our product is based on Jetson TX2 running L4T 28.1. We mainly use the video converter and encoder to convert and encode 8 video channels received on 4 MIPI inputs through V4L2. The product works in a way that after boot the user will start-stop the encoding several times. During testing we found that after repeating start-stop multiple times our system crashes.
I created a slightly modified version of the Multimedia API sample project 01_video_encode, which can be found here:
In a forever loop it repeatedly:
- starts the encoding
- encodes 250 frames
- stops the encoding
There are two ways of operation (can be changed with USE_STATIC_ENC define): static encoder allocation and dynamic allocation. The difference is that when static allocation is used, the encoder is created only once; for non-static allocation the encoder is created and deleted in every cycle. The reason behind this is that earlier I found that delete(encoder) function sometimes hangs, and this seemed to be safer.
- The encoder crashes ~270 cycles. Log: http://home.mit.bme.hu/~szanto/tegra/01_encoder_log.txt
- Memory usage during program run: http://home.mit.bme.hu/~szanto/tegra/01_mem_usage_static.txt
So memory usage increases from ~1800 MB to ~3800 MB, and goes down when the process is killed.
- The program gets killed after thousands of cycles most probably because it runs out of memory.
- Memory usage during some cycles: http://http://home.mit.bme.hu/~szanto/tegra/01_mem_usage_nonstatic.txt
Similarly to the static allocation case, memory usage definitely increases and goes back to the initial value after the process is killed.
Memory leakage is much more visible in our product:
After boot: RAM 182/7340MB (lfb 1779x4MB) cpu [0%@499,off,off,0%@500,0%@500,0%@500] EMC 0%@665 APE 150 GR3D 0%@114 During the first 8-channel encoding: RAM 2410/7340MB (lfb 1222x4MB) cpu [64%@1573,off,off,55%@1574,53%@1574,54%@1574] EMC 13%@1600 APE 150 MSENC 1113 GR3D 18%@114 Encoding stopped: RAM 1589/7340MB (lfb 1329x4MB) cpu [2%@652,off,off,0%@652,7%@653,1%@652] EMC 2%@665 APE 150 GR3D 0%@114 8-channel encoding running: RAM 2871/7340MB (lfb 1105x4MB) cpu [72%@1504,off,off,50%@1504,54%@1508,59%@1500] EMC 30%@1600 APE 150 MSENC 1113 GR3D 42%@114 Encoding stopped: RAM 2016/7340MB (lfb 1188x4MB) cpu [1%@499,off,off,0%@499,19%@499,13%@498] EMC 4%@665 APE 150 GR3D 0%@114 8-channel encoding running: RAM 3292/7340MB (lfb 1001x4MB) cpu [67%@1559,off,off,54%@1559,52%@1559,70%@1552] EMC 25%@1600 APE 150 MSENC 1113 GR3D 22%@114 Encoding stopped: RAM 2444/7340MB (lfb 1060x4MB) cpu [3%@1113,off,off,9%@1113,19%@1113,16%@1113] EMC 5%@665 APE 150 GR3D 0%@114
The issue may or may not have something in common with https://devtalk.nvidia.com/default/topic/1035330/jetson-tx2/-mmapi-r28-2-r28-1-deinitplane-of-nvvideoencoder-memory-leak-/1, but I was asked to start a new thread.
Help is appreciated.