Argus + AGX camera timeouts

Hi,
We have an application that processes 6 cameras using libargus and then runs inference on the images using libnvinfer. At a high level the camera capture happens on a series of threads (one per camera) and the inference runs on its own thread. The same application also runs on a different variant of our system that uses a Jetson Nano and 4 cameras with no issues. We are using IMX274 cameras and carrier boards from Leopard. We are using R32.4.4 of L4T and we have some driver patches for the cameras and dts files, but other than that it is pretty stock code. In the AGX system we frequently see these ‘hardware timeouts’ that effectively cause the nvargus-daemon to hang. We’ve tried this in single process mode as well (i.e. DISABLE_MULTIPROCESS=ON), and the result is the same.

An example error from the nvargus-daemon is:

Sep 23 02:56:44 argus-wingtip nvargus-daemon[11137]: SCF: Error InvalidState: (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function startCaptureInternal(), line 561)
Sep 23 02:56:44 argus-wingtip nvargus-daemon[11137]: SCF: Error InvalidState: (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCapture(), line 517)
Sep 23 02:56:44 argus-wingtip nvargus-daemon[11137]: SCF: Error InvalidState: (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 454)
Sep 23 02:56:44 argus-wingtip nvargus-daemon[11137]: SCF: Error InvalidState: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1293)
Sep 23 02:56:44 argus-wingtip nvargus-daemon[11137]: SCF: Error InvalidState: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueBubbleFillCapturesIfNeeded(), line 676)
Sep 23 02:56:44 argus-wingtip nvargus-daemon[11137]: SCF: Error InvalidState: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1135)
Sep 23 02:56:44 argus-wingtip nvargus-daemon[11137]: SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function workerThread(), line 116)
Sep 23 02:56:44 argus-wingtip nvargus-daemon[11137]: SCF: Error InvalidState: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Sep 23 02:56:53 argus-wingtip nvargus-daemon[11137]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc

We’ve tried to reproduce this timeout using various sample programs (e.g. frontend), but we’ve been unsuccessful. This only happens on our AGX system, and not on the nano system.

hello jtsd,

please narrow down the issue,
for example, could you please enable the multi-camera with the argus_camera application to preview the streams,
are you able to repo the same failure without inference?
thanks

@jtsd
Which AGX Xavier carrier board are you using? Is it LI-JXAV-MIPI-ADPT-6CAM-FP?
Is the driver also from us? If so, could you provide the name of the driver guide?

Hi Simon, yes, we are using the 6-camera LI-JXAV-MIPI-ADPT-6CAM-FP. The driver is also from you as well. We’ve been using: IMX274-MIPI_R32.2.3_Nano-CB_V1.2_20191209_Driver_Guide - we don’t have an updated version for AGX. Is there one?

Hi Jerry - we don’t see this issue using 6 cameras with the argus_camera application.