Unbound memory growth while recording videos (e.g. sample_camera_gmsl)

Autonomous Vehicles DRIVE AGX Xavier DRIVE AGX Xavier General
,
1

Hi everyone,

I am experiencing a seemingly unbound growth of memory consumption using dwSensorSerializer_serializeImageAsync to record videos in MP4 or H265 format on a DRIVE AGX. I can reproduce the behavior using the sample_camera_gmsl binary that ships with the DRIVE AGX. For example, this would be the output of tegrastats after starting the program

RAM 4246/27924MB (lfb 5663x4MB) CPU [20%,17%,8%,8%,11%,6%] GR3D_FREQ 28% AUX@38C CPU@40C Tdiode@38.25C AO@39.5C GPU@39C tj@48.5C

and this after a few hours of recording

RAM 5957/27924MB (lfb 166x4MB) CPU [15%,12%,9%,9%,7%,5%] GR3D_FREQ 52% AUX@40C CPU@42.5C Tdiode@40C AO@41C GPU@41C tj@50.5C

It doesn’t look to me like the increase of allocated memory slows down or stalls over time. While debugging this behavior I have changed the file-buffer-size that is passed to dwSensorSerializer_intialize and it looks to me like the memory consumption increases every time data is written to the file. This means, with a very large file-buffer-size the increase in memory consumption happens at a much lower rate but it continues to occur (and it does occur when the file size increases). Given that I can reproduce this behavior with an official sample application, I would think a misuse of of the Driveworks API is unlikely.

I haven’t been able to understand the root cause of this behavior and would like to know (1) if anyone else has experienced this, (2) if there is a way to prevent increasing memory allocation over time.

Best,
nwo

–

Hardware Platform: [Example: DRIVE AGX Xavier™ Developer Kit]
Software Version: [DRIVE Software 10]
Host Machine Version: [native Ubuntu 18.04]
SDK Manager Version: [1.0.1.5538]

3

Hi @nwo,

Please provide your sample_camera_gmsl command line for my reproducing here. Thanks!

4

Hi @VickNV,

Thank you for your help. I am using the following command in my experiments:

./sample_camera_gmsl --offscreen=1 --serializer-bitrate=20000000 --write-file=/tmp/video.h265

For me, memory consumption remains approximately constant without the --write-file argument. Also, varying the framerate using --serializer-framerate has an effect on how fast memory consumption grows.

Best,
nwo

6

We can observe the memory increase issue of running with writing file.
If we have any progress, we will update you here. Thanks!

7

Thank your for the status update. Looking forward to your findings!

8

Hi @VickNV,

I imagine the investigation is still ongoing, or do you have any new findings on this problem?

Best,
nwo

9

We have an internal bug to track this and will keep you posted with any new progress here. Thanks!

1 Like
10

Hi @nwo,

This is not a memory leakage issue. The increasing memory usage is due to cache. After stopping the process, you can get all the memory back by below command lines:

sudo sh -c ‘sync’
sudo sh -c ‘echo 3 > /proc/sys/vm/drop_caches’

Thanks!