Argus cam - failure to capture in multi camera context when system running resource constrained

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

I have 5 separate threads running their own capture sessions / FrameConsumer acquireFrame, pulling 4K resolution images from econ cameras. I am noticing reliability issues in acquiring new frames successfully when spinning up additional threads and processes in addition to the argus cam threads. Such errors result in logs:

SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 92)                           
SCF_AutocontrolACSync failed to wait for an earlier frame to complete.                                                                    
SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 126)             
SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 997)

Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error Timeout:  (propagating from

Are there any known issues /assumptions /requirements that these logs would reflect if these issues are observed in the context of many threads/processes running on the same cores as the argus capture sessions? Noting this issue consistently arises when using multiple (5) cameras, and is not present when using a single camera, which hints at a resource issue. Are there any flags that make argus cam more robust in this case? Are there any performance suggestions to handle this scenario? Thanks.

3

hello d.k,

may I know which JetPack release version you’re using?
could you please also narrow down the issue by adding camera numbers one-by-one,
for example, how many sources enabled together could repo this failure.

4

Hello Jerry, thanks for replying!
To clarify, this is on a Jetson Xavier NX (if you can move this thread to that category).
This is on Jetpack release 4.6.1.
This is repro with 4 cameras, we are able to receive images fine with 3 cameras and other processes running.
We see the following logs with 4 cameras:

Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function assignAllBuffersFromStream(), line 232)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/stages/CCDataSetupStage.cpp, function doHandleRequest(), line 68)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 997)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function assignAllBuffersFromStream(), line 232)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/stages/CCDataSetupStage.cpp, function doHandleRequest(), line 68)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 997)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function assignAllBuffersFromStream(), line 232)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/stages/CCDataSetupStage.cpp, function doHandleRequest(), line 68)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Oct 27 20:35:09 localhost nvargus-daemon[15628]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 997)

and

Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:32 localhost kernel: e-con_cam 34-0043: cam_stream_off 1256 MCU Get CMD Stream off Success !! 
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function allocateAndRegisterBuffers(), line 265)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function getBuffer(), line 419)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function returnFrame(), line 376)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function getBuffer(), line 438)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/CaptureContainerImpl.cpp, function assignAllBuffersFromStream(), line 232)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/stages/CCDataSetupStage.cpp, function doHandleRequest(), line 68)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 997)
Oct 27 20:38:32 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function allocateAndRegisterBuffers(), line 265)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function getBuffer(), line 419)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function returnFrame(), line 376)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function getBuffer(), line 438)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/CaptureContainerImpl.cpp, function assignAllBuffersFromStream(), line 232)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/stages/CCDataSetupStage.cpp, function doHandleRequest(), line 68)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 997)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function allocateAndRegisterBuffers(), line 265)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function getBuffer(), line 419)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function returnFrame(), line 376)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function getBuffer(), line 438)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/CaptureContainerImpl.cpp, function assignAllBuffersFromStream(), line 232)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/stages/CCDataSetupStage.cpp, function doHandleRequest(), line 68)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 997)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost nvargus-daemon[17098]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 543)
Oct 27 20:38:33 localhost kernel: e-con_cam 32-0043: cam_stream_off 1256 MCU Get CMD Stream off Success !!

and finally

Oct 27 20:38:39 localhost nvargus-daemon[17098]: waitForIdleLocked remaining request 250
Oct 27 20:38:39 localhost nvargus-daemon[17098]: waitForIdleLocked remaining request 249
Oct 27 20:38:39 localhost nvargus-daemon[17098]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
Oct 27 20:38:39 localhost nvargus-daemon[17098]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)
Oct 27 20:38:39 localhost kernel: e-con_cam 30-0043: cam_stream_off 1256 MCU Get CMD Stream off Success !! 
Oct 27 20:38:44 localhost nvargus-daemon[17098]: waitForIdleLocked remaining request 250
Oct 27 20:38:44 localhost nvargus-daemon[17098]: waitForIdleLocked remaining request 249
Oct 27 20:38:44 localhost nvargus-daemon[17098]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 927)
Oct 27 20:38:44 localhost nvargus-daemon[17098]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 893)
Oct 27 20:38:44 localhost nvargus-daemon[17098]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 406)
Oct 27 20:38:44 localhost nvargus-daemon[17098]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 506)
Oct 27 20:38:44 localhost nvargus-daemon[17098]: SCF: Error Timeout:  (propagating from src/api/CameraDriver.cpp, function deleteSession(), line 635)
Oct 27 20:38:44 localhost nvargus-daemon[17098]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 194)
Oct 27 20:38:45 localhost kernel: e-con_cam 30-0044: cam_stream_off 1256 MCU Get CMD Stream off Success !! 
Oct 27 20:38:45 localhost nvargus-daemon[17098]: === python3.6[17179]: WARNING: Cleaning up 4 outstanding streams...(Argus) Error InvalidState: Unknown stream deleted. (in src/api/CaptureSessionImpl.cpp, function outputStreamDeleted(), line 852)
Oct 27 20:38:45 localhost nvargus-daemon[17098]: (Argus) Error InvalidState: Unknown stream deleted. (in src/api/CaptureSessionImpl.cpp, function outputStreamDeleted(), line 852)
Oct 27 20:38:45 localhost nvargus-daemon[17098]: (Argus) Error InvalidState: Unknown stream deleted. (in src/api/CaptureSessionImpl.cpp, function outputStreamDeleted(), line 852)

I want to stress that stand alone on the system, the same code/app is able to capture frames with no issues, on all 5 cameras. Only when starting additional processes on the system, do we encounter the above logs. Can you provide guidance or advice on limitations of argus cam if other processes are running? Do these logs mean we are doing something incorrectly, perhaps a different flag or behavior we need to pull the images? Can you provide information as to what these logs mean, why they would be caused?

Additionally, we encounter the following log, only when the cameras “crash”:

Oct 27 21:58:23 localhost nvargus-daemon[21168]: PowerServiceCore:handleRequests: timePassed = 790

Can you provide any information as to what the PowerServiceCore log means, why it is associated with the failure to get images from the cameras, what it is indicative of?

Thank you for any help in debugging capturing images using argus library for multi camera system.

5

@d.k,

Please write your query to techsupport@e-consystems.com. Our team will solve the issue ASAP.

Sorry for the inconvenience.

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.