Memory leak in JetPack 5.1.1 when repeating the start/stop cycle of a GStreamer pipeline including nvvideoconvert and nvv4l2h264/vp9 encoders

[EduardoSalazar96]

Hi all,

I have been doing the memory leaks tests described in the Memory exhaustion when repeating the start/stop cycle of a GStreamer pipeline including nvvideoconvert and any of nvv4l2 encoders post. The followings are setup details:

• Hardware Platform (Jetson / GPU): Xavier AGX Devkit and Xavier NX Devkit
• DeepStream Version: 6.2
• JetPack Version (valid for Jetson only): 5.1.1

After running the following test:

GitHub - emi-kkobayashi/nvconv_nvenc_memory_leak: A test code to reproduce a...

I could verify that the memory increased during the test. Here are some results:

• Xavier NX and H264 encoder
  

  

• Xavier AGX and H264 encoder
  

  

As you can see in these results, the Private Memory increased during the test. I ran the test for 1 hour.

Also, find attached all the results in the following zip files:

• H264 Encoder
  test_nvconv_nvenc_memory_leak_h264_xavier_nx.zip (31.9 KB)
  test_nvconv_nvenc_memory_leak_h264_xavier_agx.zip (32.5 KB)

• VP9 Encoder
  test_nvconv_nvenc_memory_leak_vp9_xavier_nx.zip (31.2 KB)
  test_nvconv_nvenc_memory_leak_vp9_xavier_agx.zip (32.6 KB)

Questions for the NVIDIA team:

1. Could you try to reproduce the issue, please?
2. Is there a way to avoid the issue from happening?

Thanks.

[yuweiw]

How do you get the image of the memory map?
Can you try the following two scenarios？

• Change the enc to x264 enc and check if there are still memory issues
• Do not repeatedly start and stop the pipeline, just let the pipeline run a long time and check if there are still memory issues

I ran the code with DeepStream version 6.3 and there was no significant memory growth. You can also try that.

[EduardoSalazar96]

Hi @yuweiw

I used the bash script attached to this message. I used it as follows:

./mem_analysis.sh -p $(echo $(ps -a | grep nvconv_nvenc_lo) | awk '{print $1;}') -i 1 -l -g -n 3600

If the script is run as above-mentioned, it will wrap the memory values every second and will plot the results after 1 hour (3600 seconds).

I will try the two scenarios that you mentioned.

mem_analysis.sh (5.1 KB)

[yuweiw]

OK. We’ll analyze that ASAP.

[EduardoSalazar96]

Hi @yuweiw
I ran the two scenarios that you mentioned. Here are the results

Change the enc to x264 enc and check if there are still memory issues

• In the case of the Xavier NX Devkit. This is the mapped Memory, for example:
  

  
  
  I do not see any memory increase.
  Find attached below this message all the results:
  test_nvconv_nvenc_memory_leak_x264_xavier_nx.zip (36.3 KB)

• In the case of the Xavier AGX Devkit. This is the mapped Memory, for example:
  

  
  
  I do not see any memory increase either.
  Find attached below this message all the results:
  test_nvconv_nvenc_memory_leak_x264_xavier_agx.zip (33.6 KB)

Without stopping the pipeline

• In the case of the Xavier NX Devkit, I do not see any memory increase. Find attached the results below this message:
  test_nvconv_nvenc_memory_leak_not_stopping_h264_xavier_nx.zip (19.3 KB)
  test_nvconv_nvenc_memory_leak_not_stopping_vp9_xavier_nx.zip (19.9 KB)

• In the case of the Xavier AGX Devkit, I do not see any memory increase either. Find attached the results below this message:
  test_nvconv_nvenc_memory_leak_not_stopping_h264_xavier_agx.zip (19.1 KB)
  test_nvconv_nvenc_memory_leak_not_stopping_vp9_xavier_agx.zip (19.3 KB)

Please, let me know how it goes the results on your side.
Thank you!

[yuweiw]

Could you refer to the link below to use the valgrind to locate the code stack of the leak?
https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/19

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

I use the valgrind to locate the issue. Now it has no memory leak. You can refer to the picture below and wait for our future release.