IMX477 Camera Dropout Issue with Custom Carrier Board on Orin NX

user43281 · September 9, 2024, 2:31pm

We are using jetons orin NX with the following setup.

NVIDIA Orin NX
Custom carrier board
3 IMX477 Raspberry Pi HQ cameras connected via CSI ports
L4T 35.4.1 (we also tested things on 35.5.0)
Enabled resolution 4032x3040 at 30fps

We are experiencing a persistent issue where one of the IMX477 cameras drops out after a variable amount of time during recording.
The dropout occurs after as little as 10 seconds or as long as 5 hours. Initially, we saw deskew and CRC errors before the camera dropped out.

kworker/4:9-156     [004] ....   223.360804: rtcpu_vinotify_error: tstamp:7532563920 cch:3 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:125 vi_tstamp:241041604448 data:0x0000007d00e00060

Then we tested some pixel clock adjustments. After setting the pixel clock to 840MHz (previously 300MHz, the CRC errors are no longer present. However, the camera still drops out randomly with this error which was present previously too alongside the CRC error.

kworker/1:1-5731    [001] ....  7933.447653: rtcpu_vinotify_error: tstamp:248593085203 cch:2 vi:1 tag:CHANSEL_NOMATCH channel:0x04 frame:53 vi_tstamp:7954978676832 data:0x0000000000000249

I attached some logs that we captured. (The logs are after switching to 840MHz)

We’re looking for any insights or suggestions that might help resolve the camera dropout issue. Any guidance on further debugging steps or potential causes would be greatly appreciated.

Thank you for your assistance!

D1_2024-09-08_0642_.zip (12.2 MB)

JerryChang · September 10, 2024, 2:48am

hello user43281,

may I know what’s your test pipeline, for instance, are you running with v4l2 IOCTL, or gst pipeline with nvarguscamerasrc plugin?

BTW, we’ve JetPack 5.1.4 released recently, which has include some camera bug fixes, is it possible for moving to the latest release for verification?

user43281 · September 10, 2024, 6:21am

Hi @JerryChang

Thank you for your response,

We are using gstreamer pipeline.
In our main software, we use a customized nvarguscamera element, that launches all cameras with one CaptureSession (for frame synchronization purposes) , but we were able to reprduce the error with the standard nvarguscamerasrc element as well.

nvarguscamerasrc sensor-id=2 wbmode=0 ! "video/x-raw(memory:NVMM),format=(string)NV12,width=(int)4032,height=(int)3040,framerate=(fraction)29/1" ! nvvidconv ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! tee name=video1  \
nvarguscamerasrc sensor-id=1 wbmode=0 ! "video/x-raw(memory:NVMM),format=(string)NV12,width=(int)4032,height=(int)3040,framerate=(fraction)29/1" ! nvvidconv ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! tee name=video2  \
nvarguscamerasrc sensor-id=0 wbmode=0 ! "video/x-raw(memory:NVMM),format=(string)NV12,width=(int)4032,height=(int)3040,framerate=(fraction)29/1" ! nvvidconv ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! tee name=video3  \
video1. ! h264parse ! qtmux ! filesink location=video1.mp4 \
video2. ! h264parse ! qtmux ! filesink location=video2.mp4 \
video3. ! h264parse ! qtmux ! filesink location=video3.mp4

We tested 5.1.2 and 5.1.3, will try to port our changes, and test version 5.1.4 as well.

In the meantime are there other steps we can take?

Thank you very much.

user43281 · September 11, 2024, 6:25pm

Hi @JerryChang

We tested with JetPack 5.1.4, we still got the same error.

We need help as soon as possible, I’d appreciate any suggestions, or recommendations.

Thank you

user43281 · September 12, 2024, 5:56pm

Hi there.

Any response will be deeply appreciated.

Thank you very much.

Connect-Beyond-2989 · September 15, 2024, 7:36pm

@ShaneCCC & @KevinFFF @DaneLLL and @JerryChang, would love any opinions or ideas if it falls under your expertise! Thanks again 🙏

ShaneCCC · September 16, 2024, 4:17am

Please apply the attached libs and RCE firmware to verify.

camera-rtcpu-t234-rce.img.r35.5.asynchronous (519.3 KB)
libnvfusacap.so.r35.5.txt (193.6 KB)

user43281 · September 17, 2024, 11:17am

Hi @ShaneCCC,

Thank you for your repsoen,

We already switched to r35.6.0 per the suggestion of @JerryChung,

Can you send this changes for that version?

Thank you

ShaneCCC · September 17, 2024, 11:45pm

Suppose those fix are include in r35.6
Does the error are the same? How about run argus_camera?

user43281 · October 16, 2024, 8:49pm

Hi @ShaneCCC
@JerryChang
Thank you for your responses,

We did a ton of experiments in this period and even replaced our carrier board.

We discovered that the cameras are more likely to get crushed when we start them, stopping, restarting nvargusdaemon, and starting the cameras again.

It seems nvargusdaemon does not clean something,

We really need some help to get more debug info from nvargusdaemon it’s impossible to understand what’s going on.

Any help would be greatly appreciated.

ShaneCCC · October 17, 2024, 5:55am

Please get the daemon log to analysis the problem.

Thanks

user43281 · October 17, 2024, 10:33am

Thank you for your response @ShaneCCC

Here is the process for replicating the isue.

Starting our software
Then we stop it
restart it, and from that moment the cameras never return unless we reboot. It does not matter if we restart nvargusdaemon or not. (attached the logs below from nvargusaemon)

Note: This is a way we found to replicate the error, but sometimes even without any interaction the camera drops out. We get a timeout at

frames[i].reset(iFrameConsumers[i]->acquireFrame(ACQUIRE_FRAME_TIMEOUT, &frame_status));

=== NVIDIA Libargus Camera Service (0.99.33)=== Listening for connections...=== python3[4301]: Connection established (FFFF95D8D900)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
(NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 257)
(NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 61)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 112)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclStateControllerOpen: Failed ImagerGUID 3. (error 0xA000E)
NvPclOpen: PCL Open Failed. Error: 0xf
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 726)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 453)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 347)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 519)
=== python3[4301]: CameraProvider initialized (0xffff90c51120)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!





############### Up to this momment camera works fine ###################
############### Then I restart our software          ###################












PowerServiceCore:handleRequests: timePassed = 548
=== python3[4301]: CameraProvider destroyed (0xffff90c51120)(Argus) Error FileOperationFailed: Socket write failed: Broken pipe (in src/rpc/socket/common/SocketUtils.cpp, function writeSocket(), line 49)
(Argus) Error FileOperationFailed:  (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function sendResponse(), line 92)
(Argus) Error FileOperationFailed:  (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function handleRequest(), line 174)
(Argus) Error FileOperationFailed:  (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function recvThreadCore(), line 233)
(Argus) Error FileOperationFailed:  (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function run(), line 58)
=== python3[4301]: Connection closed (FFFF95D8D900)=== python3[4301]: Connection cleaned up (FFFF95D8D900)=== python3[4713]: Connection established (FFFF95D8D900)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
(NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 257)
(NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 61)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 112)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclStateControllerOpen: Failed ImagerGUID 3. (error 0xA000E)
NvPclOpen: PCL Open Failed. Error: 0xf
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 726)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 453)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 347)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 519)
=== python3[4713]: CameraProvider initialized (0xffff914a2ce0)CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/capture/FusaCaptureViCsiHw.cpp:705) (in Mutex.cpp, function lock(), line 79)
(NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/capture/FusaCaptureViCsiHw.cpp:699) (in Mutex.cpp, function lock(), line 79)
(NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/capture/FusaCaptureViCsiHw.cpp:705) (in Mutex.cpp, function lock(), line 79)
(NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/capture/FusaCaptureViCsiHw.cpp:699) (in Mutex.cpp, function lock(), line 79)
(NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
SCF_AutocontrolACSync failed to wait for an earlier frame to complete.

SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 126)
SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
SCF: Error Timeout: Sending critical error event for Session 2 
 (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 1 draining session frameStart events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 531)
SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 2 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 545)
SCF: Error InvalidState: Sensor 0 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 1 draining session frameEnd events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 648)
waitForIdleLocked remaining request 106 
waitForIdleLocked remaining request 105 
waitForIdleLocked remaining request 104 
waitForIdleLocked remaining request 103 
waitForIdleLocked remaining request 102 
waitForIdleLocked remaining request 101 
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 969)
=== python3[4713]: Connection closed (FFFF95D8D900)=== python3[4713]: There were pending client requests in the server at the time of CameraProvider destruction, and they failed to complete within a reasonable time (1500 ms). This may be due to a hardware or libargus error. Forced destruction will now proceed, which may leave the libargus server in a bad state.=== python3[4713]: WARNING: CameraProvider was not destroyed before client connection terminated.=== python3[4713]:          The client may have abnormally terminated. Destroying CameraProvider...=== python3[4713]: CameraProvider destroyed (0xffff914a2ce0)=== python3[4713]: WARNING: Cleaning up 1 outstanding requests...=== python3[4713]: WARNING: Cleaning up 1 outstanding stream settings...=== python3[4713]: WARNING: Cleaning up 1 outstanding sessions...waitForIdleLocked remaining request 106 
waitForIdleLocked remaining request 105 
waitForIdleLocked remaining request 104 
waitForIdleLocked remaining request 103 
waitForIdleLocked remaining request 102 
waitForIdleLocked remaining request 101 
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 969)

user43281 · October 17, 2024, 2:27pm

I got a bit more specific logs and details.

I modified Nvidia multimedia APIs slightly and tested with it instead of nvarguscamerasrc

here are the findings

We have 3 cameras connected, but only the camera with 0 index is causing the issue, the other two cameras work completely fine
Even for the camera index 0 one capture request succeeds, after the first request on the argus lib side there are no errors it still says the request succeded, but I see in the logs of nvargus-daemon that the request failed with the following message
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734) (Argus) Error InvalidState: (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)

here are the full logs of nvargus-dameon in the process of running a modified version of userAlternatingAutoExposure,

I want to emphasize again that after reboot it works completely fine, but if I stop and start the executable it starts this behavior.

=== NVIDIA Libargus Camera Service (0.99.33)=== Listening for connections...=== test_controller[6528]: Connection established (FFFF850FA900)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
(NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 257)
(NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 61)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 112)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclStateControllerOpen: Failed ImagerGUID 3. (error 0xA000E)
NvPclOpen: PCL Open Failed. Error: 0xf
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 726)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 453)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 347)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 519)
=== test_controller[6528]: CameraProvider initialized (0xffff80bcc0c0)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!





SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 1 draining session frameStart events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 531)
SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 2 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 545)
PowerServiceCore:handleRequests: timePassed = 4515
SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 3 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 545)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 1 draining session frameEnd events 3
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 648)
SCF: Error InvalidState: Sensor 0 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 2 draining session frameEnd events 2
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 648)
SCF: Error Timeout: Sending critical error event for Session 0 
 (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
SCF: Error InvalidState: Sensor 0 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 3 draining session frameEnd events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 648)
SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
SCF: Error Timeout: NvRmSyncWait failed (in src/api/Buffer.cpp, function cpuWaitFences(), line 622)
SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function cpuWaitInputFences(), line 543)
SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function acquire(), line 680)
SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function ScopedBufferLock(), line 657)
PowerServiceCore:handleRequests: timePassed = 4996
SCF: Error InvalidState: Sensor 0 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 4 draining session frameEnd events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 648)
SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 4 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 545)
SCF: Error InvalidState: Sensor 0 already in same state 
 (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 5 draining session frameEnd events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 648)
SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 5 continue draining session frameStart events 1
 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 545)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)

ShaneCCC · October 18, 2024, 4:43am

Both of the camera are IMX477?
Please verify the basic capture function by v4l2-ctl.

v4l2-ctl --stream-mmap -c bypass_mode=0

user43281 · October 18, 2024, 10:23am

Yes, all cameras are IMX477 connected with CSI ports.

The command below crushes the board, and it fully reboots.

For basic capturing, from the test version of userAlternatingAutoExposure I can verify that the first capture request succeeds, I am getting a single image from there with IFrameConsumer, right after the first frame comes nvargus-daemon starts printing errors and no response comes.

ShaneCCC · October 18, 2024, 4:55pm

If v4l2-ctl unable capture normally that could be HW or sensor driver have problem.

user43281 · October 18, 2024, 4:57pm

Hi @ShaneCCC

Any ways to get a bit more info which part exactly is causing the crush?

ShaneCCC · October 18, 2024, 5:01pm

Get the trace log to check if more detail.

JerryChang · October 21, 2024, 1:50am

hello user43281,

is it caused by kernel panic?
you may try setting up a terminal to keep gathering kernel logs, i.e. $ dmesg --follow.

anyways, here’re patches to fix kernel panic when VI trying to recover camera stream.
please see-also Topic 305007 to apply the fixes for verification.

user43281 · October 22, 2024, 12:07pm

Hi @JerryChang

Thank you for your response,

I see two patches int he topic you linked,

Are those patches included in the 35.6.0 or I should apply them?

Topic		Replies	Views
Problem with IMX477 on 32.7.5 in slave mode. (bug in nvarguscamerasrc?) Jetson Nano camera , gstreamer	37	124	August 27, 2024
Nvargus crashes with unreliable CSI camera connections on Jetpack 5.1.2 Jetson Orin NX camera , nvbugs	15	1687	December 14, 2023
Infinite timeout support Jetson Orin NX camera	22	1173	March 5, 2024
Deepstream application fails after runnig rtsp for a while Jetson Orin Nano rtsp	16	785	April 18, 2024
Video stream crashes in jetpack 5.1.1 xavier nx Jetson Xavier NX camera	15	880	December 25, 2023
Nvarguscamerasrc Fails to Stream Camera Jetson Orin NX camera	2	29	November 14, 2024
iBufferOutputStream acquireBuffer Timeout dosn't fire Jetson Xavier NX camera	17	928	August 17, 2022
Camera problems using CSI MIPI camera on Jetson TX2 Jetson TX2 camera	6	1100	October 18, 2021
120fps with IMX219 and binned modes Jetson Xavier NX camera	49	3516	January 18, 2023
iBufferOutputStream Segfault while reading capture metadata Jetson Xavier NX camera	15	739	August 24, 2022

IMX477 Camera Dropout Issue with Custom Carrier Board on Orin NX

Related topics