How to troubleshoot camera "SCF: Error InvalidState: Timeout!"

Deqing_S · September 27, 2024, 8:27pm

We have a system of orin nano on a leopard imaging carrier board. with 3 of the DS90UB954 FPD-link serializer, and connect to 6 of the DS90UB953 serializer+AR0234 sensor cameras.

The same system worked OK when we have I2C command in driver to start and stop video streaming on sensors. As a part of the development, we removed I2C command to start and stop streaming and kept the camera streaming to the
jetson. And we found some random error when using Argus API to pause and restart the camera.

At the beginning we used

iCaptureSession->stopRepeat();
iCaptureSession->waitForIdle(); //about 600ms for the 2 commands
iCaptureSession->cancelRequests();

to pause the camera and

iCaptureSession->repeat(requestPtr);

to restart the capture. The error happens about 1 in 4 starts.

Then we tried to clear everything including the Argus::UniqueObj<Argus::CaptureSession> captureSession; It happens 1 in 15~20 times.

How would you suggest to troubleshoot it? How can I see lower level logs of the camera?

Thanks and the following is the error.

SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 281466386776067 draining session frameStart events 281466386776066
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 281466386776068 continue draining session frameStart events 281466386776065
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
SCF: Error InvalidState: Sensor 0 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 3 draining session frameEnd events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error InvalidState: Sensor 0 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 4 draining session frameEnd events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)

ShaneCCC · September 30, 2024, 3:52am

Do this sequence.

iCaptureSession->cancelRequests();
iCaptureSession->stopRepeat();
iCaptureSession->waitForIdle();

Deqing_S · September 30, 2024, 4:09pm

Thank you for your reply.

I tried to change the stop order but I still get the issue, here I’ve attached 3 errors in 3 sessions on 3 different cameras. But, in some case it did help a lot (explain later).

SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 1 draining session frameEnd events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error InvalidState: Sensor 2 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 2 draining session frameEnd events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error Timeout: Sending critical error event for Session 2 
 (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
SCF: Error InvalidState: Sensor 2 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 2, capture sequence ID = 1 draining session frameStart events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
SCF: Error InvalidState: Sensor GUID 2 is in error state. Skipping requests, capture sequence ID = 2 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)


SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 1, capture sequence ID = 281466386776065 draining session frameStart events 281466386776066
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
SCF: Error InvalidState: Sensor GUID 1 is in error state. Skipping requests, capture sequence ID = 281466386776066 continue draining session frameStart events 281466386776065
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
SCF: Error InvalidState: Sensor 1 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 1, capture sequence ID = 1 draining session frameEnd events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error InvalidState: Sensor GUID 1 is in error state. Skipping requests, capture sequence ID = 281466386776067 continue draining session frameStart events 281466386776064
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
Segmentation fault (core dumped)


SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 5, capture sequence ID = 1 draining session frameStart events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
SCF: Error InvalidState: Sensor GUID 5 is in error state. Skipping requests, capture sequence ID = 2 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
SCF: Error InvalidState: Sensor 5 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 5, capture sequence ID = 1 draining session frameEnd events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error InvalidState: Sensor 5 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 5, capture sequence ID = 2 draining session frameEnd events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
SCF: Error Timeout: Sending critical error event for Session 5 
 (in src/api/Session.cpp, function sendErrorEvent(), line 1039)

Also, if I load a driver that send out start/stop streaming I2C command to sensor with the same Argus code, it runs perfectly well, at least for the 40 restart cycle I did today with 6 cameras.

I made the following test with 6 cameras. I did have microcontroller on camera to initialize sensor so the MIPI stream can start automatically without Jetson intervention,

Streaming type	crash test
Linux driver send I2C start/stop streaming I2C signal	no crash for at least 40 restarts
Linux driver not send I2C start/stop streaming I2C signal, sensor keep streaming	crash for around 3~6 restarts
Linux driver not send I2C start/stop streaming I2C signal, having mosfet cut power after stop, and restore power 500ms after start	crash for around 2~3 restarts
Linux driver not send I2C start/stop streaming I2C signal, having mosfet cut power after stop, and restore power 1000ms after start	crash for around 4~7 restarts
Linux driver not send I2C start/stop streaming I2C signal, having mosfet cut power after stop, and restore power 1500ms after start	no crash for at least 30 restarts
Linux driver not send I2C start/stop streaming I2C signal, having mosfet cut power after stop, and restore power 1500ms after start, before changing the “cancelRequests” order	crash for around 2~3 restarts

In a summary, by changing the order of “cancelRequests”, and power on camera late enough, the argus can work. Although there is a lot of expected (NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 793) due to i2c communication failure, it can work.

Without checking the detail of Argus lib, I think I can make a assumption that when the argus is starting camera, we should make sure there is no video streaming. Is that true? Or there is a better way to evaluate it?

Also, in some of our previous test, when we try to start the camera, it seems Argus just queue our request and return immediately. Judging from the I2C signals, the real opening process can happen hundreds of milliseconds later. Is there a way to check if the Argus actually opens the camera, so we can power on the camera ASAP?

ShaneCCC · October 1, 2024, 5:23am

What’s the BSP version?

Try below command run argus_daemon as infinite waiting mode.

sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon

Deqing_S · October 1, 2024, 2:47pm

I got

sudo apt-cache show nvidia-jetpack

Package: nvidia-jetpack
Version: 5.1.3-b29

Also, sudo enableCamInfiniteTimeout=1 nvargus-daemon doesn’t seem help

SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 1, capture sequence ID = 3 draining session frameEnd events 2

ShaneCCC · October 2, 2024, 5:15am

Replace the libnvscf.so by attached file to run the infinite daemon to try again.

libnvscf.so.r35.5.inifite.txt (8.4 MB)