I got an issue that when I ran my app for a while, the nvagus-daemon would get errors and from then on all the frames I got from my app Stayed the same as the moment the issue happened.
The issue I got from nvargus-daemon is shown below:
Dec 08 19:47:06 cp-copy17-0094 nvargus-daemon[3664]: Module_id 30 Severity 2 : (fusa) Error: InvalidState propagating from:/capture/src/fusaViHandler.cpp 759
Dec 08 19:47:06 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 600)
Dec 08 19:47:06 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Dec 08 19:47:06 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
Dec 08 19:47:08 cp-copy17-0094 nvargus-daemon[3664]: Module_id 30 Severity 2 : (fusa) Error: Timeout propagating from:/capture/src/fusaIspHandler.cpp 811
Dec 08 19:47:08 cp-copy17-0094 nvargus-daemon[3664]: Module_id 30 Severity 2 : (fusa) Error: Timeout propagating from:/capture/src/fusaIspHandler.cpp 778
Dec 08 19:47:08 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: FuSa Capture Status failed! (in src/services/capture/FusaCaptureIspHw.cpp, function waitIspFrameEnd(), line 365)
Dec 08 19:47:08 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: ISP request error with status code 0 (in src/services/capture/FusaCaptureIspHw.cpp, function waitIspFrameEnd(), line 381)
Dec 08 19:47:09 cp-copy17-0094 nvargus-daemon[3664]: Module_id 30 Severity 2 : (fusa) Error: Timeout propagating from:/capture/src/fusaIspHandler.cpp 811
Dec 08 19:47:09 cp-copy17-0094 nvargus-daemon[3664]: Module_id 30 Severity 2 : (fusa) Error: Timeout propagating from:/capture/src/fusaIspHandler.cpp 778
Dec 08 19:47:09 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: FuSa Capture Status failed! (in src/services/capture/FusaCaptureIspHw.cpp, function waitIspFrameEnd(), line 365)
Dec 08 19:47:09 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: ISP request error with status code 0 (in src/services/capture/FusaCaptureIspHw.cpp, function waitIspFrameEnd(), line 381)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 78379 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 514)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: Timeout waiting on frame start sensor guid 1, capture sequence ID = 234895 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 514)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 600)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
Dec 08 19:47:11 cp-copy17-0094 nvargus-daemon[3664]: Module_id 30 Severity 2 : (fusa) Error: Timeout propagating from:/capture/src/fusaViHandler.cpp 776
Dec 08 19:47:12 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
Dec 08 19:47:12 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
Dec 08 19:47:12 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
Dec 08 19:47:12 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: NvRmSyncWait failed (in src/api/Buffer.cpp, function cpuWaitFences(), line 622)
Dec 08 19:47:12 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function cpuWaitInputFences(), line 543)
Dec 08 19:47:12 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function acquire(), line 680)
Dec 08 19:47:12 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function ScopedBufferLock(), line 657)
Dec 08 19:47:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: NvRmSyncWait failed (in src/api/Buffer.cpp, function cpuWaitFences(), line 622)
Dec 08 19:47:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function cpuWaitInputFences(), line 543)
Dec 08 19:47:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function acquire(), line 680)
Dec 08 19:47:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function ScopedBufferLock(), line 657)
Dec 08 19:47:15 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 78352
Dec 08 19:47:15 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 78351
Dec 08 19:47:15 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 78353
Dec 08 19:47:15 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
Dec 08 19:47:15 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Session.cpp, function abortCaptures(), line 888)
Dec 08 19:47:15 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Dec 08 19:47:15 cp-copy17-0094 nvargus-daemon[3664]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Dec 08 19:47:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: NvRmSyncWait failed (in src/api/Buffer.cpp, function cpuWaitFences(), line 622)
Dec 08 19:47:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function unassignFromCapture(), line 387)
Dec 08 19:47:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/CaptureContainerImpl.cpp, function releaseBuffer(), line 579)
Dec 08 19:47:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/ac_stages/AfAnalysisStage.cpp, function pushStatsToAmr(), line 166)
Dec 08 19:47:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/ac_stages/AfAnalysisStage.cpp, function pushResultsToAmr(), line 193)
Dec 08 19:47:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/ac_stages/AfAnalysisStage.cpp, function doHandleRequest(), line 268)
Dec 08 19:47:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
Dec 08 19:47:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
Dec 08 19:47:18 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: NvRmSyncWait failed (in src/api/Buffer.cpp, function cpuWaitFences(), line 622)
Dec 08 19:47:18 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function unassignFromCapture(), line 387)
Dec 08 19:47:18 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/CaptureContainerImpl.cpp, function releaseBuffer(), line 579)
Dec 08 19:47:18 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/ac_stages/AfAnalysisStage.cpp, function pushStatsToAmr(), line 166)
Dec 08 19:47:18 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/ac_stages/AfAnalysisStage.cpp, function pushResultsToAmr(), line 193)
Dec 08 19:47:18 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/ac_stages/AfAnalysisStage.cpp, function doHandleRequest(), line 268)
Dec 08 19:47:18 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
Dec 08 19:47:18 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
Dec 08 19:48:10 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: 4 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
Dec 08 19:48:10 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: 2 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 300)
Dec 08 19:48:10 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 49)
Dec 08 19:48:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: NvRmSyncWait failed (in src/api/Buffer.cpp, function cpuWaitFences(), line 622)
Dec 08 19:48:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function cpuWaitInputFences(), line 543)
Dec 08 19:48:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function acquire(), line 680)
Dec 08 19:48:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function ScopedBufferLock(), line 657)
Dec 08 19:48:14 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 49)
Dec 08 19:48:15 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 78352
Dec 08 19:48:15 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 78353
Dec 08 19:48:15 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
Dec 08 19:48:15 cp-copy17-0094 nvargus-daemon[3664]: (Argus) Error Timeout: (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 216)
Dec 08 19:48:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
Dec 08 19:48:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
Dec 08 19:48:17 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
Dec 08 19:48:19 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 230411
Dec 08 19:48:19 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 230412
Dec 08 19:48:19 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 230410
Dec 08 19:48:19 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 230413
Dec 08 19:48:19 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
Dec 08 19:48:19 cp-copy17-0094 nvargus-daemon[3664]: (Argus) Error Timeout: (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 216)
Dec 08 19:48:20 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 78352
Dec 08 19:48:20 cp-copy17-0094 nvargus-daemon[3664]: waitForIdleLocked remaining request 78353
Dec 08 19:48:20 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
Dec 08 19:48:20 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Session.cpp, function abortCaptures(), line 888)
Dec 08 19:48:20 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/api/Session.cpp, function shutdown(), line 407)
Dec 08 19:48:22 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
Dec 08 19:48:22 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
Dec 08 19:48:22 cp-copy17-0094 nvargus-daemon[3664]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
I have boost the clock with:
sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
But it doesn’t make any difference
Anything I can do for further debugging and to get more information about this issue?