Camera Bringup - AGX ORIN Industrial - FMON_NVCSI error

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
,
1

Hello Everyone,

I’m trying to run a camera on a custom board with the AGX Orin Industrial version. It works well on the commercial version within the devkit using a similar device tree and driver, but it’s not working on the industrial version with custom board. I’m using JetPack 36.4.

I am getting logs from FSI, but I’m not sure what they mean and can’t find any information on them. What could I be missing?

Serial port:

verify_rate_range : FMON_NVCSI: rate 10045312 below min 18750000
fmon_update_config: FMON_NVCSI: detected fault 0x80

LOG TRACE:

 CaptureSchedule-5927    [004] .......   180.918578: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
 CaptureSchedule-5927    [004] .......   180.918582: csi_s_power: enable : 0x1
 CaptureSchedule-5927    [004] .......   180.919605: tegra_channel_set_stream: enable : 0x1
 CaptureSchedule-5927    [004] .......   180.920367: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
 CaptureSchedule-5927    [004] .......   180.920371: csi_s_stream: enable : 0x1
 CaptureSchedule-5927    [004] .......   180.920372: tegra_channel_set_stream: imx678 1-001a : 0x1
 CaptureSchedule-5927    [004] .......   181.181281: vi_task_submit: class_id:48 ch:1 syncpt_id:49 syncpt_thresh:0 pid:5927 tid:1012
 CaptureSchedule-5927    [004] .......   181.197153: vi_task_submit: class_id:48 ch:1 syncpt_id:49 syncpt_thresh:0 pid:5927 tid:1012
 CaptureSchedule-5927    [004] .......   181.197301: vi_task_submit: class_id:48 ch:1 syncpt_id:49 syncpt_thresh:0 pid:5927 tid:1012
     kworker/5:1-81      [005] .......   181.197565: rtcpu_nvcsi_intr: tstamp:7164234948 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x10000000
     kworker/5:1-81      [005] .......   181.197568: rtcpu_nvcsi_intr: tstamp:7164235442 class:GLOBAL type:PHY_INTR0 phy:0 cil:1 st:0 vc:0 status:0x00000008
     kworker/5:1-81      [005] .......   181.197573: rtcpu_vinotify_event: tstamp:7164805818 cch:1 vi:0 tag:VIFALC_TDSTATE channel:0x0b frame:0 vi_tstamp:229273548256 data:0x359cc80010000000
     kworker/5:1-81      [005] .......   181.197574: rtcpu_vinotify_event: tstamp:7164805957 cch:1 vi:0 tag:VIFALC_TDSTATE channel:0x0b frame:0 vi_tstamp:229273553600 data:0x0000000031000001
     kworker/5:1-81      [005] .......   181.197575: rtcpu_vinotify_event: tstamp:7164806110 cch:1 vi:0 tag:VIFALC_ACTIONLST channel:0x0b frame:0 vi_tstamp:229273556128 data:0x0000000007020001
     kworker/5:1-81      [005] .......   181.253567: rtcpu_vinotify_event: tstamp:7165431789 cch:1 vi:0 tag:VIFALC_TDSTATE channel:0x0b frame:0 vi_tstamp:229289409152 data:0x359cc50010000000
     kworker/5:1-81      [005] .......   181.253570: rtcpu_vinotify_event: tstamp:7165431953 cch:1 vi:0 tag:VIFALC_TDSTATE channel:0x0b frame:0 vi_tstamp:229289414560 data:0x0000000031000002
 CaptureSchedule-5927    [006] .......   185.826129: tegra_channel_set_stream: enable : 0x0
 CaptureSchedule-5927    [006] .......   185.826134: tegra_channel_set_stream: imx678 1-001a : 0x0
     kworker/5:1-81      [005] .......   185.873539: rtcpu_nvcsi_intr: tstamp:7310929860 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000001
     kworker/5:1-81      [005] .......   185.873542: rtcpu_nvcsi_intr: tstamp:7310929860 class:GLOBAL type:PHY_INTR1 phy:0 cil:0 st:0 vc:0 status:0x00000125
     kworker/5:1-81      [005] .......   185.873542: rtcpu_nvcsi_intr: tstamp:7310929860 class:GLOBAL type:PHY_INTR1 phy:0 cil:1 st:0 vc:0 status:0x00000104
 CaptureSchedule-5927    [006] .......   185.901547: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x0
 CaptureSchedule-5927    [006] .......   185.901553: csi_s_stream: enable : 0x0```

DMESG:

[ 1071.581926] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[ 1071.581951] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[ 1071.583561] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel


Thanks in advance
4

nvarguscamerasrc throws timeout:

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
Error generated. gstnvarguscamerasrc.cpp, threadExecute:734 NvBufSurfaceFromFd Failed.
Error generated. gstnvarguscamerasrc.cpp, threadFunction:245 (propagating)

nvargus-daemon:

SCF: Error InvalidState: Sensor 0 already in same state
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 0 draining session frameEnd events 3
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: SCF: Error InvalidState: Sensor 0 already in same state
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 1 draining session frameEnd events 2
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 1 continue draining session frameStart events 2
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: SCF: Error InvalidState: Sensor 0 already in same state
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 2 draining session frameEnd events 1
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
Feb 27 17:58:11 jetson-orin-ind nvargus-daemon[1142]: Module_id 30 Severity 2 : (fusa) Error: Timeout propagating from:/capture/src/fusaViHandler.cpp 820

extended nvarus-daemon:

 SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 281470681743360 draining session frameStart events 15199644447408131
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]:  (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 281470681743361 continue draining session frameStart events 15199644447408130
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]:  (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: SCF: Error InvalidState: Sensor GUID 0 is in error state. Skipping requests, capture sequence ID = 281470681743362 continue draining session frameStart events 15199644447408129
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]:  (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: FiberScheduler: cc 101, session 0, fiber 0xffff44000b70 aborted in async operation
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: SCF: Error InvalidState: Sensor 0 already in same state
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]:  (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 281470681743360 draining session frameEnd events 281470681743363
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]:  (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: returnBuffer: apiInput 0 and bufferIndex 0 and buffer 0xffff4c009910 and stream 0xffff4c001360 and instream (nil)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: Created fiber 0xffff44000fc0 for CC 103 globalID 103 session 0
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: CaptureService cancelling request 1
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: Session 0 aborting capture 103 (API ID: 3)FiberScheduler: cc 102, session 0, fiber 0xffff44000e20 aborted in async operation
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: returnBuffer: apiInput 0 and bufferIndex 0 and buffer 0xffff4c00b0d0 and stream 0xffff4c001360 and instream (nil)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: Session 0 aborting capture 102 (API ID: 2)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: Session 0 aborting capture 101 (API ID: 1)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: Thread 7 is working on CC 103 session 0 globalID 103 step 0, fiber: 0xffff44000fc0
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: Fiber 0xffff44000fc0 is aborting in CC 103 Session 0
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: FiberScheduler: cc 103 session 0, fiber 0xffff44000fc0 aborted
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: FiberScheduler: fiber 0xffff44000b70 exiting
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: FiberScheduler: fiber 0xffff44000e20 exiting
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: NvPclSettingsUpdate: Sending Updated Settings through PCL
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: NvPclSettingsApply: Applying last settings through PCL
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: SCF: Error InvalidState: Sensor 0 already in same state
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]:  (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: disposing CC 102 Session 0
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 0, capture sequence ID = 281470681743361 draining session frameEnd events 281470681743362
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]:  (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: apply:+++++++++++++++++++++++
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: returnBuffer: apiInput 0 and bufferIndex 4 and buffer 0xffff3001e010 and stream (nil) and instream (nil)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: returnBuffer: apiInput 0 and bufferIndex 5 and buffer 0xffff3001f0b0 and stream (nil) and instream (nil)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: disposing CC 101 Session 0
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: returnBuffer: apiInput 0 and bufferIndex 4 and buffer 0xffff3802b600 and stream (nil) and instream (nil)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: returnBuffer: apiInput 0 and bufferIndex 5 and buffer 0xffff3802c6a0 and stream (nil) and instream (nil)
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: Thread 8 is waiting
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: FiberScheduler: fiber 0xffff44000fc0 exiting
Feb 27 18:02:44 jetson-orin-ind nvargus-daemon[9476]: Thread 9 is waitingSCF: Error InvalidState: Sensor 0 already in same state
6

The error tell the data/clock lane control have problem. Looks like the sensor output signal have problem.
Please probe the signal to confirm.

     kworker/5:1-81      [005] .......   185.873539: rtcpu_nvcsi_intr: tstamp:7310929860 class:GLOBAL type:PHY_INTR0 phy:0 cil:0 st:0 vc:0 status:0x00000001
     kworker/5:1-81      [005] .......   185.873542: rtcpu_nvcsi_intr: tstamp:7310929860 class:GLOBAL type:PHY_INTR1 phy:0 cil:0 st:0 vc:0 status:0x00000125
     kworker/5:1-81      [005] .......   185.873542: rtcpu_nvcsi_intr: tstamp:7310929860 class:GLOBAL type:PHY_INTR1 phy:0 cil:1 st:0 vc:0 status:0x00000104
 CaptureSchedule-5927    [006] .......   185.901547: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x0
1 Like
7

Thank you, that makes sense. The camera started working in 2-lane mode. I’ll test the other 2 lanes to determine what might be wrong with them.

9

What about those logs from FSI? Do you know if this is something important, or can it be ignored?

10

Suppose it’s sensor driver message.
I don’t have idea about it.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.