Argus Error: Freezing Streams with Multiple Cameras

Hello,
We are using an application to run the streams of multiple cameras ~30fps. This application uses Argus library.
We are currently facing the issue of the freeze when running multiple cameras (without stressing the system) after around 2 hours, note that the freeze issue does not appear if we use one camera. We get the following error:

We are globally facing the same issue as here : Argus image acquisition crashes after a few days , the iCaptureSession->isRepeating() fails.
We use:

  • Jetson AGX Orin
  • Jetpack 5.1.3 / L4T 35.5.0
  • Argus library 0.99.3.3

Thanks

hello ikram.el.moussaoui,

may I also confirm how many cameras in your system, how about its resolution, bits-per-pixel?
in addition, had you also test with v4l2-ctl to Verifying the V4L2 Sensor Driver.

Hello @JerryChang, Thank you for the answer.
We are using 2 cameras in our system :

  • First camera : captures images with a resolution of 3848x2168 and a bit depth of 12 bits per pixel.

  • Second camera : captures images with a resolution of 1936x1552 and also has a bit depth of 12 bits per pixel.

For the driver side, we made sure that everthing works fine.
Note that the freeze happens after around 2 hours from launching the 2 streams at the same time.
Thank you

hello ikram.el.moussaoui,

may I know which application you’re used, are you testing with argus_camera sample app?
is it possible to test with MMAPI sample, i.e. 13_argus_multi_camera?

FYI, we’ve tested two 4K@30-fps streaming (although they’re 10-bit sensors), and it’s passed long during testing.
anyways, if you still seeing the failure, could you please also give it a try to enable infinite timeout for verificaton.
for instance,
$ sudo service nvargus-daemon stop
$ sudo enableCamInfiniteTimeout=1 nvargus-daemon
$ ./argus_camera

Hello @JerryChang,
Thank you for your response. Regarding the application we utilize, it’s an internal tool developed, leveraging LibArgus for stream retrieval and display.

We conducted testing on the provided sample under two distinct conditions:

  • First Test: Under stress test/power mode 15W, we encountered a freeze in both cameras after a few minutes from initiating the streams. The error encountered is depicted below:

  • Second test: In the absence of a stress test using power mode MAXN, we observed no freezing even after running for over 15 hours.

In our application workflow, for each frame we :

            createNvBuffer
            
            NvBufSurfaceFromFd
            
            Map the NvBufSurface for CPU access using : NvBufSurfaceMap
            
            Synchronize the NvBufSurface for CPU using: NvBufSurfaceSyncForCpu
            
            Retrieve the buffers
            
            Synchronize the NvBufSurface for device using: NvBufSurfaceSyncForDevice
            
            Unmap the NvBufSurface using : NvBufSurfaceUnMap
            
            Destroy the NvBufSurface using : NvBufSurfaceDestroy

On the producer side, we utilize “captureRequest” instead of “capture” for each frame, employing one thread per camera (producer/consumer). This approach contrasts with the sample 13, which manages all cameras within a single thread.
Do you have any suggestions or proposed fixes based on the explanations above ?
Thank you

hello ikram.el.moussaoui,

Argus support both (a) one thread per camera, and (b) all cameras within a single thread.
did you have camera use-case with system stressed? it’s suggest to test with real use-case instead of a dummy service to fully occupy the CPU/GPU resources.

Hello @JerryChang,
Thank you for your response. In terms of the stress test, we use it only to expedite error detection. Typically, errors may take around 3 or 4 hours to manifest in real-case usage scenarios. We do not have a camera use-case with a stressed system.
Thanks again.

hello ikram.el.moussaoui,

how about give it another try with expedite error detection by using power mode MAXN as well.

Hello @JerryChang,
Thank your answer.
The issue is solved from the libs provided here : SegFault argus

hello ikram.el.moussaoui,

let’s figure out what’s actually happened.
don’t you use the latest Jetpack release version, i.e. Jetpack 5.1.3 / L4T 35.5.0?