Memory leak in camera reading using Argus

Hasa · October 17, 2024, 3:29am

I am using the Jetson MMAPI sample 13, to receive images from a camera. The code works fine, until the end of the frame capturing progress.
When the code wants to finish the capture request, there are a few lines of code as follows:

for (uint32_t i = 0; i < streamCount; i++)
{
        ICaptureSession *iCaptureSession =
            interface_cast<ICaptureSession>(captureHolders[i].get()->getSession());
        iCaptureSession->waitForIdle();
}
/* Destroy the capture resources */
for (uint32_t i = 0; i < streamCount; i++)
{
      captureHolders[i].reset();
}
/* Wait for the rendering thread to complete */
PROPAGATE_ERROR(consumerThread.shutdown());
/* Shut down Argus */
g_cameraProvider.reset();
/* Cleanup EGL Renderer */
delete g_renderer;

This code stucks at captureHolders[i].reset(); for a very long time until it throws the timeout error. I have found if I comment the iCaptureSession->waitForIdle(); part and send some extra triggers to the camera at the final stage, this object gets released and the whole ConsumerThread object and CaptureHolder will be released.

However my main program is open and I need to make instances of these two classes multiple times (I need to access to the camera with multiple resolution change so I need to close all the argus-related objects and open again). When I do that with the changes in the code that I mentioned, I get a memory leakage. The amount of leakage is about 3 times of the image size at each iteration. What can be the reason of the leak?

I checked all the allocations and deallocations in the sample 13 and my code exactly mimic them, so I am suspecting the closing part.

ShaneCCC · October 17, 2024, 6:08am

Please reference to argus_camera for changing the sensor resolution process.

Thanks

Hasa · October 17, 2024, 6:16am

The 09_argus_camera_jpeg you mean? Or jetson_multimedia_api/argus/apps/camera?
If you mean argus/apps/camera, which part of it should check?
Thank you

ShaneCCC · October 17, 2024, 7:33am

Yes, it’s argus/apps/camera you can run it to change sensor mode and check if any memory leak and check the code follow to compare.

Thanks

Hasa · October 17, 2024, 8:07am

Thanks, I will check that one as well, but the behaviour of the sample 13 is a point of question. Is it expected to have trouble closing this app (sample 13)? I don’t understand the reason behind the long waiting before the captureHolder reset.

And since the needed deallocations are done:

if (m_compositedFrame)
{
        NvBufSurf::NvDestroy(m_compositedFrame);
        m_compositedFrame = 0;
}

for (uint32_t i = 0; i < m_streams.size(); i++)
{
    if (m_dmabufs[i]) {
         NvBufSurf::NvDestroy(m_dmabufs[i]);
         m_dmabufs[i] = 0;
    }
}

Wheresles can we expect the memory leak?

Hasa · October 21, 2024, 8:45am

@ShaneCCC @DaneLLL @kayccc Sorry to mention again, any idea about the above question? This is highly critical for our case, thank you.

ShaneCCC · October 23, 2024, 5:49am

Does the sensor streaming off before waitForIdle()? Suppose the sensor should streaming to have driver to consume the capture request to avoid timeout.

Hasa · October 23, 2024, 6:17am

@ShaneCCC
Can you elaborate a bit? You mean the trigger stream? If so then yes, the trigger is continuing until all camera related objects will be released.
In the software side, the Consumer thread is also up in that time, (waiting for leftover frames I guess). If I wait a very long time, the consumer will throw the timout error and be released and the program will be closed finally, but in real case, not possible to wait this much.

ShaneCCC · October 24, 2024, 7:14am

Suppose the waitForIdle() is waiting all capture request in the queue if the sensor streaming early may cause the timeout.

Hasa · October 24, 2024, 8:15am

Still didn’t get it. The sensor streaming late can be related, but how can it stream early when we are trying to finish?
I didn’t understand anything from the answers.

ShaneCCC · October 25, 2024, 3:38am

Does this problem only happen on multiple cameras?

Hasa · October 28, 2024, 2:48am

No, I tried with 1 Camera and 2 Cameras, both were the same.

ShaneCCC · October 28, 2024, 4:36am

I didn’t see the problem with 13_argus_multi_camera sample code on JP5.x
What’s your release?

Hasa · October 28, 2024, 6:08am

This is my environment:

Model: NVIDIA Jetson Xavier NX Developer Kit - Jetpack 5.1 [L4T 35.2.1]
Hardware:
 - Module: NVIDIA Jetson Xavier NX (16GB ram)
Platform:
 - Distribution: Ubuntu 20.04 focal
 - Release: 5.10.104
Libraries:
 - CUDA: 11.4.315
 - cuDNN: 8.6.0.166

Can you share your test code as a stress test on sample 13? The original sample doesn’t repeat the initialization and closing. It only runs it one time.

ShaneCCC · October 28, 2024, 8:16am

Yes, I run the original sample one time.
How do you stress test it?

Hasa · October 28, 2024, 8:21am

I mean just running everything in a loop (initial->capture->destroy) to see the potential memory leak. But anyway even for one time run I still stuck at captureHolders[i].reset();
Is there anything out of the sample code that I can check to see everythins is working fine?

ShaneCCC · October 28, 2024, 8:32am

What’s the sensor?
You may check the argus-daemon log. Run below command the start the test.

journalctl -u nvargus-daemon -f

Hasa · October 29, 2024, 1:53am

The sensor is Sony IMX531

And here is the log of the command, while I was running Sample13:

journalctl -u nvargus-daemon -f
– Logs begin at Thu 2024-10-24 15:51:57 JST. –
Oct 29 10:44:57 s02v4dev2 nvargus-daemon[516794]: D/ libnvphs:socket: Warning: connecting to Power Hinting Service failed. Is PHS running?
Oct 29 10:44:57 s02v4dev2 nvargus-daemon[516794]: === argus_multi_camera[516880]: CameraProvider initialized (0xffff886e26c0)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
Oct 29 10:44:57 s02v4dev2 nvargus-daemon[516794]: E/ libnvphs:socket: Error[2]: socket connection /var/lib/nvphs/nvphsd.ctl to PHS failed: No such file or directory
Oct 29 10:44:57 s02v4dev2 nvargus-daemon[516794]: D/ libnvphs:socket: Warning: connecting to Power Hinting Service failed. Is PHS running?
Oct 29 10:44:57 s02v4dev2 nvargus-daemon[516794]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=MinCPU, value=4294967295, timeout_ms=1000]: queue_or_send() failed
Oct 29 10:44:59 s02v4dev2 nvargus-daemon[516794]: CAM: serial no file already exists, skips storing againCAM: serial no file already exists, skips storing againPowerServiceCore:handleRequests: timePassed = 1398
Oct 29 10:46:03 s02v4dev2 systemd[1]: Stopping Argus daemon…
Oct 29 10:46:03 s02v4dev2 systemd[1]: nvargus-daemon.service: Succeeded.
Oct 29 10:46:03 s02v4dev2 systemd[1]: Stopped Argus daemon.
Oct 29 10:46:03 s02v4dev2 systemd[1]: Started Argus daemon.
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: === NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections…=== argus_multi_camera[517189]: Connection established (FFFFAE281900)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: NvPclHwGetModuleList: No module data found
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: NvPclHwGetModuleList: No module data found
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Oct 29 10:46:14 s02v4dev2 nvargus-daemon[517139]: NvPclHwGetModuleList: No module data found
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: CAM: serial no file already exists, skips storing again---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: CAM: serial no file already exists, skips storing again---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: (NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: (NvCamV4l2) Error ModuleNotPresent: (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: (NvOdmDevice) Error ModuleNotPresent: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 107)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: NvPclStartPlatformDrivers: Failed to start module drivers
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: NvPclStateControllerOpen: Failed ImagerGUID 2. (error 0xA000E)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: NvPclOpen: PCL Open Failed. Error: 0xf
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 708)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: SCF: Error BadParameter: (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 453)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: SCF: Error BadParameter: (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 333)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: SCF: Error BadParameter: (propagating from src/api/CameraDriver.cpp, function getSource(), line 505)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: E/ libnvphs:socket: Error[2]: socket connection /var/lib/nvphs/nvphsd.ctl to PHS failed: No such file or directory
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: D/ libnvphs:socket: Warning: connecting to Power Hinting Service failed. Is PHS running?
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: === argus_multi_camera[517189]: CameraProvider initialized (0xffffa86af110)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: E/ libnvphs:socket: Error[2]: socket connection /var/lib/nvphs/nvphsd.ctl to PHS failed: No such file or directory
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: D/ libnvphs:socket: Warning: connecting to Power Hinting Service failed. Is PHS running?
Oct 29 10:46:15 s02v4dev2 nvargus-daemon[517139]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=MinCPU, value=4294967295, timeout_ms=1000]: queue_or_send() failed
Oct 29 10:46:17 s02v4dev2 nvargus-daemon[517139]: CAM: serial no file already exists, skips storing againCAM: serial no file already exists, skips storing againPowerServiceCore:handleRequests: timePassed = 1403

And this is the situation of the sample executive:

ShaneCCC · October 29, 2024, 4:55am

Could you upgrade to r35.6 to verify.

Thanks

Hasa · October 29, 2024, 5:01am

Alright. Only MMAPI or the whole Jetpack?

Topic		Replies	Views
How to make Argus in Jetson 35.2.1 recover after a corrupted frame? Jetson AGX Orin camera , nvbugs	30	2463	July 7, 2023
Argus API is not stable in JetPack 5 and gstreamer doesnt work Jetson AGX Xavier camera , gstreamer	11	990	May 30, 2023
nvargus-daemon crash on simultaneous use of Argus camera and USB camera Jetson TX2	13	7690	October 18, 2021
Video stream crashes in jetpack 5.1.1 xavier nx Jetson Xavier NX camera	15	907	December 25, 2023
Camera problems using CSI MIPI camera on Jetson TX2 Jetson TX2 camera	6	1117	October 18, 2021
Argus errors on some boots Jetson TX2 camera , gstreamer	8	956	March 28, 2023
iBufferOutputStream Segfault while reading capture metadata Jetson Xavier NX camera	15	744	August 24, 2022
Nvargus-daemon hanging - Jetson Nano 32.7.3 Jetson Nano camera , gstreamer	17	146	August 11, 2024
Nvargusdaemon crashing after camera has been active for a short time Jetson Xavier NX camera , gstreamer	4	1124	January 11, 2023
Camera crashes nvargus-daemon Jetson Nano camera	10	1284	August 10, 2022

Memory leak in camera reading using Argus

Related topics