libArgus AcquireFrame for multiple cameras fails after a few seconds

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

I am trying to write a CUDA camera consumer.
I tested the camera setup with argus_camera and I am able to see 4 live camera images simultaniously with MultiSession.

In order to get the scheduling right I based my application on argus_camera’s MultiSession Task with Composer and Dispatcher and I replaced all GL calles with cuda calls and I do something like: (pseudocode)

cuEGLStreamConsumerConnect(&m_cudaConnection, m_eglStream);

threadExecute() {
while(1) {
  for (m_sessions) {
    eglQueryStreamKHR(display, m_eglStream, EGL_STREAM_STATE_KHR, &m_streamState)
    if (m_streamState == EGL_STREAM_STATE_NEW_FRAME_AVAILABLE_KHR
        && m_state != CUDA_PROCESSING) {
      cuEGLStreamConsumerAcquireFrame( &m_cudaConnection, &m_cudaResource, &m_cudaStream, kAcquireTimeout);
      cuGraphicsResourceGetMappedEglFrame(&m_eglFrame, m_cudaResource, 0, 0);
      cuTexObjectCreate(&m_tex, &resDesc, &texDesc, NULL);
    }
  }
}

getTextures() {
    m_state = CUDA_PROCESSING;
    return std::vector<CUtexObject> m_textures;
}

cudaFinished() {
    m_state = CONNECTED;
    cuTexObjectDestroy(m_tex);
    cuEGLStreamConsumerReleaseFrame(&m_cudaConnection, m_cudaResource,
}

I am processing the textures from a different thread that starts the cuda kernel. It periodically gets the mapped textures from the other thread. if cuda is finished it instructs the acquire thread to continue acquireing for all Sessions

The acquire thread works at first and it succesfully acquires and maps the resources to textures.
After a few seconds the argus daemon crashes with:

Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: NvRmChannelSubmit: NvError_IoctlFailed with error code 22
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 29, SyncPointValue = 0)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error IoctlFailed:  (propagating from src/services/capture/CaptureServiceDeviceIsp.cpp, function startCapture(), line 536)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error IoctlFailed:  (propagating from src/services/capture/CaptureServiceCore.cpp, function doMEMtoISPCapture(), line 528)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error IoctlFailed:  (propagating from src/services/capture/CaptureServiceCore.cpp, function issueCapture(), line 368)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error IoctlFailed:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1213)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error IoctlFailed:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1043)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 591)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 358)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error IoctlFailed:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Jul 19 14:08:28 tegra-ubuntu argus_daemon[18922]: SCF: Error IoctlFailed: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 131)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 92)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 92)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 92)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 92)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 126)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 126)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 126)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 126)
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Jul 19 14:08:33 tegra-ubuntu argus_daemon[18922]: SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)

I don’t know how to handle this. this might be a race condition or locking issue???!!
Do I need to always create and destroy the Texture?

2

hello stfl,

[url]https://developer.nvidia.com/embedded/jetpack[/url]
we had some performance improvements to solve acquire frame failure with multi-session use-case.
suggest you moving to the latest public release JetPack-4.2.1 (l4t-r32.2) and have verification.
thanks

3

Hi Stfl,
To understand CUDA Consumer, Producer and Argus interoperability I would suggest you to check cudaBayerDemosaic and cudaHistogram samples in tegra_multimedia_api/argus/samples directory. For multiple camera operation kindly check the syncSensor and multiSensor Sample.

Personal recommendation,
Create two output streams with reference to multiSensor sample, Check syncSensor, cudaHistogram and cudaBayerDemosaic for creating CUDA Consumer on the above created streams.

4

Hi We are experiencing the same problem.
Our setup is 3 capture session connected to 3 cuda threads for color processing and a preview thread to view cuda processed frames.

Strange thing is,
it works with 3 camera setup with 2MP resolution.
it works with 2 camera setup with 8MP resolution.
it works with 3 camera setup with 8MP resolution until the exactly 50th frame.

At the 50th frame even cuEGLStreamConsumerAcquireFrame function call returns with success, series of error happens like stfl’a entry.

we are using jetpack 4.2 after testing it with jetpack 4.2.1, i will let you know results…

Regards.

5

we tested on Jetpack-4.2.2 (l4t-32.2.1) and now it works.
Regards.