Argus errors on TX2

Hello. We are experiencing some problems with the nvargus daemon when using a CSI MIPI camera from Leopard Imaging (LI-IMX274-MIPI-M12) together with GStreamer on our Jetson TX2 (using a Spacely carrier board from ConnectTech).

The camera works most of the time, but sometimes it looks like the driver goes into an error state and we are not longer able to capture images, even after restarting the whole system.

The only way to get the camera working again is to physically disconnect and reconnect the cable from the carrier board.

See below the output from the system log when trying to use nvgstcapture to capture images from the camera:

-- Logs begin at Sat 2018-02-03 18:09:28 CET. --
feb. 03 18:09:36 scout-onboard-3 systemd[1]: Started Argus daemon.
feb. 03 18:10:14 scout-onboard-3 nvargus-daemon[6763]: === NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...=== nvgstcapture[8001]: Connection established (7F91A871D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
feb. 03 18:10:14 scout-onboard-3 nvargus-daemon[6763]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
feb. 03 18:10:14 scout-onboard-3 nvargus-daemon[6763]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
feb. 03 18:10:14 scout-onboard-3 nvargus-daemon[6763]: NvPclHwGetModuleList: No module data found
feb. 03 18:10:15 scout-onboard-3 nvargus-daemon[6763]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
feb. 03 18:10:15 scout-onboard-3 nvargus-daemon[6763]: ---- imager: No override file found. ----
feb. 03 18:10:15 scout-onboard-3 nvargus-daemon[6763]: ---- imager: No override file found. ----
feb. 03 18:10:15 scout-onboard-3 nvargus-daemon[6763]: ---- imager: No override file found. ----
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]: === nvgstcapture[8001]: CameraProvider initialized (0x7f8c8c3e50)NvViErrorDecode Stream 0.0 failed: ts 63565076064 frame 0 error 2 data 0x00400060
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]: NvViErrorDecode CaptureError: CsimuxFrameError (2)
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]: NvViErrorDecode See https://wiki.nvidia.com/wmpwiki/index.php/Camera_Debugging/CaptureError_debugging for more information and links to documents.
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]: CsimuxFrameError_Regular : 0x00400060
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:     Stream ID                [ 2: 0]: 0
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:         
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:     VPR state from fuse block    [ 3]: 0
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:         
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:     Frame end (FE)              [ 5]: 1
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:         A frame end has been found on a regular mode stream.
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:     CSI_FAULT                   [ 6]: 1
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:         An FE packet was found and marked with a CSI Error
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:     CSI_CODE                 [25:20]: 0x4
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]:         Check the CSI specification for the meaning. See https://wiki.nvidia.com/wmpwiki/index.php/Camera_Debugging/CaptureError_debugging#CSI_CODE
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]: captureErrorCallback Stream 0.0 capture 2 failed: ts 63565076064 frame 0 error 2 data 0x00400060
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]: SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 630)
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]: SCF: Error BadValue:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
feb. 03 18:10:16 scout-onboard-3 nvargus-daemon[6763]: SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
feb. 03 18:10:17 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout: ISP port 0 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 478)
feb. 03 18:10:25 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
feb. 03 18:10:25 scout-onboard-3 nvargus-daemon[6763]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 561)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState:  (propagating from src/services/capture/NvViCsiHw.cpp, function startCapture(), line 508)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState:  (propagating from src/services/capture/DeviceRecordNv.cpp, function doCSItoISPCapture(), line 110)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoISPCapture(), line 547)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 460)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1276)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1107)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error BadParameter: Fiber not present (in src/components/CaptureContainerImpl.cpp, function detachFiber(), line 597)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error BadParameter: CC not being processed (in src/api/Session.cpp, function processingEnding(), line 1080)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error InvalidState: Active fiber not found (in src/components/FiberScheduler.cpp, function endOfLife(), line 548)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 637)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 358)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 637)
feb. 03 18:10:26 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 358)
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 637)
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 358)
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: SCF: Error OverFlow:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function checkCaptureComplete(), line 1198)
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: SCF: Error OverFlow:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: SCF: Error OverFlow: Worker thread CaptureScheduler checkFramePending failed (in src/common/Utils.cpp, function workerThread(), line 133)
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: launchCC abort cc 105 session 0
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function capture(), line 834)
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: (Argus) Error Timeout:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
feb. 03 18:10:27 scout-onboard-3 nvargus-daemon[6763]: (Argus) Error InvalidState: MetadataResult callback for unknown capture. (in src/api/CaptureSessionImpl.cpp, function metadataResult(), line 906)
feb. 03 18:10:28 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout: ISP port 0 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 478)
feb. 03 18:10:28 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 561)
feb. 03 18:10:28 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)
feb. 03 18:10:28 scout-onboard-3 nvargus-daemon[6763]: SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
feb. 03 18:10:29 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout: ISP port 1 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 501)
feb. 03 18:10:31 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout: ISP port 2 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 512)
feb. 03 18:10:31 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 519)
feb. 03 18:10:31 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
feb. 03 18:10:31 scout-onboard-3 nvargus-daemon[6763]: SCF: Error Timeout: Worker thread IspHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)

It does not help to restart nvargus-daemon.

Is this a known problem and how may we proceed to debug and resolve it?

Thanks in advance,
-Øystein

hello Oystein1,

suggest you should also contact with carrier board vendor to check if there’s hardware issues.

moreover,
this failure means VI/CSI driver cannot receive sensor signaling correctly,
could you please share some details,
i.e. which JetPack release you’re working with, please also share the gstreamer pipelines for reference.

We are using JetPack 4.3. Our GStreamer pipeline is quite complex and is set up via the GStreamer C API. But we have been able to reproduce the problem by using only the nvgstcapture sample.

hello Oystein1,

please access Approaches for Validating and Testing the V4L2 Driver, please also have a try with v4l2 standard controls to check the camera basic functionality.
thanks