Nvargus-daemon: InvalidState Status syncpoint signaled

[thienvu.daonguyen]

I have the same problem with below topic on Jetson Orin Nano Dev Kit 4GB. And I used Camera IMX 296 on it.
So the pipeline of gstreamer I use for my application which is: (both camera on the two pipelines have only difference with sensor-id=0-1).
“nvarguscamerasrc sensor-id=0 wbmode=1 aeantibanding=1 ee-mode=1 tnr-mode=1 tnr-strength=-1 saturation=1.0 aelock=0 awblock=0 exposurecompensation=0.0 name=video_src ! nvvidconv ! videoflip name=video_flip method=0 ! nvvidconv ! video/x-raw, format=I420, width=1440, height=1080 ! tee name=t t. ! queue max-size-buffers=2 leaky=2 ! interpipesink name=camera_src1 forward-eos=true forward-events=true async=true sync=false t. ! queue max-size-buffers=30 leaky=2 ! appsink name=raw_app_sink emit-signals=true async=false drop=true max-buffers=30”

Intermittent camera failure on Orin

Hi, This is a follow up to I reproduced camera error with off-the-shelf LI-JETSON-IMX274-DUAL sensors attached to Orin as well as our Omnivision cameras. For IMX274 sensors I am using driver and DTB provided for Nvidia. The error happened approximately once per 50 hours for IMX274 - always with the second camera. But with our Omnivision camera it happens more often: approximately once per 10 hours with one Orin/cameras set and once per 1 hour on another. Bottom line is that camera always …

When the error occurred on nvargus-daemon service, so following that my application could not get frames from nvarguscamerasrc elements. So please help me resolve this problems.
I also tried to boost all the VI/CSI/ISP clocks. But It still got errors/

[ShaneCCC]

How long to hit the problem?
How about run single camera individually?

[thienvu.daonguyen]

It ran about 7 hours and then hit the problem (two camera).
I have not tested with single camera. But it’ll take long time to do it.

[ShaneCCC]

It’s looks like stability issue. I would suggest to narrow down the root cause.
Like run single camera for each camera and verify two camera by v4l2-ctl by script simultaneously.

v4l2-ctl --stream-mmap -c bypass_mode=0 -d /dev/video0 
v4l2-ctl --stream-mmap -c bypass_mode=0 -d /dev/video1

[thienvu.daonguyen]

You mean that I should run the two commands simultaneously. And then check it get error or not?. I’ll do it.
But I think it could be issue about nvargus-daemon service. So that nvarguscamerasrc element can not get frames and then lead to break my app. You want to figure out the cause with v4l2-ctl if it also hit the problem, so the problem is on my camera, right?.

[ShaneCCC]

I mean run argus with single camera for each or them instead simultaneously.
And Yes v4l2-ctl to verify the sensor driver and HW.

[thienvu.daonguyen]

I’ll do it. But if there is not any problems with these commands, what shoud i do?
Because I think it will take long time for checking it.

[ShaneCCC]

You also can try simple pipeline to clarify if any element cause the problem.

gst-launch-1.0 nvarguscamerasrc  ! nvvidconv ! fpsdisplaysink video-sink=fakesink --verbose

[thienvu.daonguyen]

I ran “gst-launch-1.0 nvarguscamerasrc sensor-id=0-1 ! nvvidconv ! x264enc ! h264parse ! fpsdisplaysink video-sink=fakesink --verbose”. And then It got this error as below.
Please review it and give me your help. Thanks.

ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: UNAVAILABLE
Additional debug info:
Argus Error Status
CONSUMER: ERROR OCCURRED
Execution ended after 0:43:19.217698840
Setting pipeline to NULL …
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch() , line 91)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch() , line 91)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch() , line 91)
GST_ARGUS: Cleaning up
Freeing pipeline …
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch() , line 91)
(Argus) Error InvalidState: Argus client is exiting with 4 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 366).

[thienvu.daonguyen]

And I had got error from nvargusdaemon-service before I hit the problem.

Module_id 30 Severity 2 : (fusa) Error: InvalidState Status syncpoint signaled but status value not updated in:/capt>
Module_id 30 Severity 2 : (fusa) Error: InvalidState propagating from:/capture/src/fusaViHandler.cpp 759
SCF: Error InvalidState: (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(),>
SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
SCF: Error InvalidState: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(>
Module_id 30 Severity 2 : (fusa) Error: ResourceAlreadyInUse All captures are already pending, no idle captures avai>
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function startCaptu>
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCaptu>
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(),>
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCap>
SCF: Error ResourceAlreadyInUse: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCap>
SCF: Error ResourceAlreadyInUse: (propagating from src/common/Utils.cpp, function workerThread(), line 114)
SCF: Error ResourceAlreadyInUse: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function>
Module_id 30 Severity 2 : (fusa) Error: ResourceAlreadyInUse propagating from:/capture/src/fusaViHandler.cpp 475
SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 644)
SCF: Error Timeout: (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 426)
SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function a>
SCF: Error InvalidState: (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
SCF: Error InvalidState: (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequ>
SCF: Error InvalidState: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function a>
SCF: Error InvalidState: (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
SCF: Error InvalidState: (propagating from src/components/stages/SensorCaptureStage.cpp, function doHandleRequest()>
SCF: Error InvalidState: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
SCF: Error Timeout: (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
SCF: Error Timeout: (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), >
Module_id 30 Severity 2 : (fusa) Error: ResourceAlreadyInUse propagating from:/capture/src/fusaViHandler.cpp 475
SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 644)
SCF: Error Timeout: (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 426)
SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function a>
SCF: Error InvalidState: (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
SCF: Error InvalidState: (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequ>
SCF: Error InvalidState: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function a>
SCF: Error InvalidState: (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function a>
SCF: Error InvalidState: (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 411)
SCF: Error InvalidState: (propagating from src/components/stages/SensorCaptureStage.cpp, function doHandleRequest()>
SCF: Error InvalidState: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
SCF: Error Timeout: (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
SCF_AutocontrolACSync failed to wait for an earlier frame to complete.
SCF: Error Timeout: (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), >
SCF: Error Timeout: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
SCF: Error InvalidState: 3 buffers still pending during EGLStreamProducer destruction (propagating from src/services>
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), li>
SCF: Error InvalidState: 6 buffers still pending during EGLStreamProducer destruction (propagating from src/services>
SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), li>
waitForIdleLocked remaining request 77882
waitForIdleLocked remaining request 77881
waitForIdleLocked remaining request 77880
waitForIdleLocked remaining request 77879
waitForIdleLocked remaining request 77878
waitForIdleLocked remaining request 77877
waitForIdleLocked remaining request 77876
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
(Argus) Error Timeout: (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 216)
waitForIdleLocked remaining request 88200
waitForIdleLocked remaining request 88196
waitForIdleLocked remaining request 88197
waitForIdleLocked remaining request 88195
waitForIdleLocked remaining request 88194
waitForIdleLocked remaining request 88193
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 922)
(Argus) Error Timeout: (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 216)

[thienvu.daonguyen]

@ShaneCCC Can you help me this issue?

[ShaneCCC]

Those message tell unable get frame data from sensor.
Please make sure your the sensor output signal and driver stability.

[thienvu.daonguyen]

The nvargus-daemon service was not get frame data from sensor. After that I restarted the nvargus-daemon service and then It got frame data from sensor normally.
So I think this issue is not on the sensor output signal.

[ShaneCCC]

So it’s stability issue instead of functionality problem?
What’s the fail rate?
Did you boost the clocks to try.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

[thienvu.daonguyen]

I mentioned about boosting clocks
“I also tried to boost all the VI/CSI/ISP clocks. But It still got errors.”

[ShaneCCC]

I still suspect it’s sensor stability cause the problem.
How many time to reproduce the problem?
Did you verify the sensor HW driver by v4l2-ctl?

[thienvu.daonguyen]

@ShaneCCC
Running with
"v4l2-ctl --stream-mmap -c bypass_mode=0 -d /dev/video0 "
"v4l2-ctl --stream-mmap -c bypass_mode=0 -d /dev/video1 "
I have run these command for a day. It seems not to occur the issue like nvargus.
The time to reproduce the problem is about 7-8 hours. (nvargus)

[ShaneCCC]

What’s the verison?

cat /etc/nv_tegra_release

[thienvu.daonguyen]

@ShaneCCC Here is the version I’m using.

R35 (release), REVISION: 3.1, GCID: 32827747, BOARD: t186ref, EABI: aarch64, DATE: Sun Mar 19 15:19:21 UTC 2023

[ShaneCCC]

Did you try simple pipeline without “x264enc ! h264parse” to narrow down it?