Leak on libargus

[andres.artavia]

Hi,
Recently we made some testing on JetPack 5.1.2, and noticed a constant decrease of Available memory when leaving capture pipelines going. Happened on two different sets of hardware

• Orin NX, with Jetpack 5.1.2 with a set of 2 Mipi OX08B40 cameras
• Xavier NX, with Jetpack 5.1.2 with a set of 4 imx219 cameras with serdes(ub960/ub953)

Testing and results for Orin NX

Pipelines used:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! fakesink silent=false nvarguscamerasrc sensor-id=1 ! fakesink silent=false 

After some time we can see this:

Notes:

• nvarguscamerasrc is default, without any custom changes.

Testing and results for Xavier NX
That behaviour is also observed on a completely different set of hardware, in this case we were able to test using the libargus sample at /usr/src/jetson_multimedia_api/samples/10_argus_camera_recording
With the following patch on main.cpp, to capture indefinitely:

653c653
-     for (int i = 0; i < CAPTURE_TIME && !frameConsumerThread.isInError(); i++)
---
+     while (!frameConsumerThread.isInError())

Command used:

./argus_camera_recording -i 0 -f out_s0.h264 & ./argus_camera_recording -i 1 -f out_s1.h264 & ./argus_camera_recording -i 2 -f out_s2.h264 & ./argus_camera_recording -i 3 -f out_s3.h264

We can see a similar trend as the Orin:

PS. The memAvailable number is taken from /proc/memAvailable, but verified with top and jtop

Regards,
Andres
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.c om/
Website: www.ridgerun.com

[DaneLLL]

Hi,
Thanks for reporting it. We will reproduce the issue and check further.

[DaneLLL]

Hi,
For information, do you observe the issue in launching single camera like:

$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! fakesink

or

./argus_camera_recording -i 0 -f /dev/null

The commands you shared are for launching multiple cameras. Not sure if you observe the issue while single camera is being launched.

[andres.artavia]

Sure, Im going to take a look, give me a couple of hours to leave the test going. Will post the findings one I have them.

Regards,
Andres

[andres.artavia]

Hi, @DaneLLL
Same behavior, left it going around 50min on the xavier nx, with:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! fakesink

Let me know if you need me to test something else.

Regards,
Andres

[DaneLLL]

Hi,
We got the result in overngiht run:
Start running

$ cat /proc/meminfo | grep MemAvailable
MemAvailable:   14134344 kB

After 7 hours:

# cat /proc/meminfo | grep MemAvailable
MemAvailable:   14033272 kB

After overnight (about 23 hours):

# cat /proc/meminfo | grep MemAvailable
MemAvailable:   14004400 kB

Looks like the avavilable memory is not linearly decreased. Could you try and see if you have same observation?

[andres.artavia]

Hi,
Sorry for the late reply, I can see that its not linear, there are some kind of steps.

Regards,
Andres

[andres.artavia]

Hi, @DaneLLL , any updates on that?

[DaneLLL]

Hi,
We are not able to hit OOM killer in overnight test. Would need some time to check further. Not sure if you have tried 3-day and 5-day test and still see the memory gets increased? It is helpful if you have more information.

[andres.artavia]

Sure, I think I can spare the hw and leave it going for some days. Will post results next week

[DaneLLL]

Hi,
Would be great if you can run argus_camera app to launch the cameras and see if the issue is present. To clarify if it is specific to running gstreamer command.

[jcaballero]

Hi,

We tested with the argus_camera_recording sample app (/usr/src/jetson_multimedia_api/samples/10_argus_camera_recording) and we also see a decrease in available memory in time. We ran two tests:

Test 1:
Number of cameras: 4
Resolution: 1280x720@30fps
Duration: 21 hours
Decrease in memory available: 83.7 MB

Test 2:
Number of cameras: 4
Resolution: 1280x720@30fps
Duration: 12.5 hours
Decrease in memory available: 44.1 MB

In your post from September 5th, you reported a decrease on the available memory from 14134344 kB to 14004400 kB (129.9 MB) in the 23 hour test.

Could there be a small/slow memory leak?

[DaneLLL]

Hi,
There is fix for Argus stack in Jetpack 5.1.3. Please upgrade and check again.