ISP timeout on TX2 making precise triggering impossible

medical-imaging-9 · October 17, 2023, 11:58am

Hello,
We are using a TX2 connected to an off the shelf carrier board, with some sony industrial global shutter sensors.

We would like to send a trigger pulse to our system, which would allow us (with only a few microseconds of latency) to capture an image at a specific time.

However we are not able to do this because when the ISP does not receive any data for some short period of time, there is a timeout and the whole system crashes. This requires us to feed the ISP with a constant stream of images so that it doesn’t time out, and then we grab the “nearest” frame to the one we want. This is not workable for us.

My questions:

I guess that on TX2 this is simply the way it is, and we can’t solve it?
if this is the case, would we have better luck using e.g. AGX Orin instead?

Let me stress that it is absolutely a requirement to have extremely low latency triggering, and running the sensor at e.g. 30FPS and “grabbing the nearest frame” is absolutely NOT an option.

thank you
John

JerryChang · October 18, 2023, 2:05am

hello medical-imaging-9,

may I know which Jetpack release version you’re working with.
could you please also share the logs for reference? you may enable a terminal for running with $ dmesg --follow to keep gathering the logs.

furthermore,
please also give it a try with v4l2 standard IOCTL to fetch the camera stream.

medical-imaging-9 · October 20, 2023, 12:24pm

We are using L4T 32.4.2 which should correspond to JetPack 4.4 Developer Preview.

We tried to call the method Buffer* Argus::IBufferOutputStream::acquireBuffer with the camera sensors turned off in order to emulate the setup with the IMX camera sensors used in the precise triggering mode that are not sending any frames yet. Even though we used Argus::TIMEOUT_INFINITE as the first argument of acquireBuffer method, we ended up with our recorder application crashing after 1.22 second due to the waitCsiFrameEnd timeout:

1970-01-01T00:00:42.603304+0000 buildroot recorder[620]: cameras.cpp:886:wait_for_sync_pulse(): 2: calling acquireBuffer
1970-01-01T00:00:42.603304+0000 buildroot recorder[620]: cameras.cpp:91:get_sensor_modes(): sensor modes count 2
1970-01-01T00:00:42.603472+0000 buildroot recorder[620]: cameras.cpp:1401:start_capture(): Starting repeat capture requests.
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: Error: waitCsiFrameEnd timeout guid 1
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI Stream Id = 2 Virtual Channel = 0
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: **VI Debug Registers
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_CSIMUX_STAT_FRAME_8 = 0x0000001b
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_CSIMUX_FRAME_STATUS_0 = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_CFG_INTERRUPT_STATUS_0 = 0x3f000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_ISPBUFA_ERROR_0 = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_FMLITE_ERROR_0 = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_NOTIFY_ERROR_0 = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: CSI Stream Id = 2 Brick Id = 1
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: **CSI Debug Registers
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: CILA_INTR_STATUS_CILA[0x20400] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: CILB_INTR_STATUS_CILB[0x20c00] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: INTR_STATUS[0x200a4] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERR_INTR_STATUS[0x200ac] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC0[0x20094] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC1[0x20098] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC2[0x2009c] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC3[0x200a0] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: SCF: Error BadValue: (propagating from src/common/Utils.cpp, function workerThread(), line 116)
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: Error: waitCsiFrameStart timeout guid 1
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI Stream Id = 2 Virtual Channel = 0
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: **VI Debug Registers
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_CSIMUX_STAT_FRAME_8 = 0x0000001b
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_CSIMUX_FRAME_STATUS_0 = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_CFG_INTERRUPT_STATUS_0 = 0x3f000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_ISPBUFA_ERROR_0 = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_FMLITE_ERROR_0 = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: VI_NOTIFY_ERROR_0 = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: CSI Stream Id = 2 Brick Id = 1
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: **CSI Debug Registers
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: CILA_INTR_STATUS_CILA[0x20400] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: CILB_INTR_STATUS_CILB[0x20c00] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: INTR_STATUS[0x200a4] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERR_INTR_STATUS[0x200ac] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC0[0x20094] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC1[0x20098] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC2[0x2009c] = 0x00000000
1970-01-01T00:00:43.822212+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC3[0x200a0] = 0x00000000
1970-01-01T00:00:43.824120+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:43.824120+0000 buildroot recorder[620]: SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
1970-01-01T00:00:43.824120+0000 buildroot recorder[620]: SCF: Error BadValue: (propagating from src/common/Utils.cpp, function workerThread(), line 116)
1970-01-01T00:00:43.824120+0000 buildroot recorder[620]: SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: Error: waitCsiFrameEnd timeout guid 0
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI Stream Id = 4 Virtual Channel = 0
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: **VI Debug Registers
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_CSIMUX_STAT_FRAME_16 = 0x00000012
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_CSIMUX_FRAME_STATUS_0 = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_CFG_INTERRUPT_STATUS_0 = 0x3f000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_ISPBUFA_ERROR_0 = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_FMLITE_ERROR_0 = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_NOTIFY_ERROR_0 = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: CSI Stream Id = 4 Brick Id = 2
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: **CSI Debug Registers
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: CILA_INTR_STATUS_CILA[0x30400] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: CILB_INTR_STATUS_CILB[0x30c00] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: INTR_STATUS[0x300a4] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERR_INTR_STATUS[0x300ac] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC0[0x30094] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC1[0x30098] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC2[0x3009c] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC3[0x300a0] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: SCF: Error BadValue: (propagating from src/common/Utils.cpp, function workerThread(), line 116)
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: Error: waitCsiFrameStart timeout guid 0
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI Stream Id = 4 Virtual Channel = 0
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: **VI Debug Registers
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_CSIMUX_STAT_FRAME_16 = 0x00000012
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_CSIMUX_FRAME_STATUS_0 = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_CFG_INTERRUPT_STATUS_0 = 0x3f000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_ISPBUFA_ERROR_0 = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_FMLITE_ERROR_0 = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: VI_NOTIFY_ERROR_0 = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: CSI Stream Id = 4 Brick Id = 2
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: **CSI Debug Registers
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: CILA_INTR_STATUS_CILA[0x30400] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: CILB_INTR_STATUS_CILB[0x30c00] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: INTR_STATUS[0x300a4] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERR_INTR_STATUS[0x300ac] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC0[0x30094] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC1[0x30098] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC2[0x3009c] = 0x00000000
1970-01-01T00:00:44.013425+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC3[0x300a0] = 0x00000000
1970-01-01T00:00:44.014753+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:44.014753+0000 buildroot recorder[620]: SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
1970-01-01T00:00:44.014753+0000 buildroot recorder[620]: SCF: Error BadValue: (propagating from src/common/Utils.cpp, function workerThread(), line 116)
1970-01-01T00:00:44.014753+0000 buildroot recorder[620]: SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: Error: waitCsiFrameStart timeout guid 2
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI Stream Id = 0 Virtual Channel = 0
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: **VI Debug Registers
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_CSIMUX_STAT_FRAME_0 = 0x00000009
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_CSIMUX_FRAME_STATUS_0 = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_CFG_INTERRUPT_STATUS_0 = 0x3f000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_ISPBUFA_ERROR_0 = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_FMLITE_ERROR_0 = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_NOTIFY_ERROR_0 = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: CSI Stream Id = 0 Brick Id = 0
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: **CSI Debug Registers
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: CILA_INTR_STATUS_CILA[0x10400] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: CILB_INTR_STATUS_CILB[0x10c00] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: INTR_STATUS[0x100a4] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERR_INTR_STATUS[0x100ac] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC0[0x10094] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC1[0x10098] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC2[0x1009c] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC3[0x100a0] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameStart(), line 624)
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: SCF: Error BadValue: (propagating from src/common/Utils.cpp, function workerThread(), line 116)
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: SCF: Error BadValue: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: Error: waitCsiFrameEnd timeout guid 2
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI Stream Id = 0 Virtual Channel = 0
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: **VI Debug Registers
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_CSIMUX_STAT_FRAME_0 = 0x00000009
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_CSIMUX_FRAME_STATUS_0 = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_CFG_INTERRUPT_STATUS_0 = 0x3f000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_ISPBUFA_ERROR_0 = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_FMLITE_ERROR_0 = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: VI_NOTIFY_ERROR_0 = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: CSI Stream Id = 0 Brick Id = 0
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: **CSI Debug Registers
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: CILA_INTR_STATUS_CILA[0x10400] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: CILB_INTR_STATUS_CILB[0x10c00] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: INTR_STATUS[0x100a4] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERR_INTR_STATUS[0x100ac] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC0[0x10094] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC1[0x10098] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC2[0x1009c] = 0x00000000
1970-01-01T00:00:44.204891+0000 buildroot recorder[620]: ERROR_STATUS2VI_VC3[0x100a0] = 0x00000000
1970-01-01T00:00:44.206296+0000 buildroot recorder[620]: *****************************************
1970-01-01T00:00:44.206296+0000 buildroot recorder[620]: SCF: Error BadValue: timestamp cannot be 0 (in src/services/capture/NvViCsiHw.cpp, function waitCsiFrameEnd(), line 711)
1970-01-01T00:00:44.206296+0000 buildroot recorder[620]: SCF: Error BadValue: (propagating from src/common/Utils.cpp, function workerThread(), line 116)
1970-01-01T00:00:44.206296+0000 buildroot recorder[620]: SCF: Error BadValue: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)

The larger dmesg log (kernel log) corresponding to the system crash is attached. How can we overcome this issue and make the acquireBuffer method robust in the no-frame-present scenario?
dmesg_log.txt (115.9 KB)

JerryChang · October 23, 2023, 2:16am

hello medical-imaging-9,

note, it’s not suggest to use “Developer Preview” version for development.
could you please try moving to JetPack 4.6.4.

according to the logs,
it looks like VI engine cannot receive camera frames. also, it’s reporting an error with timestamp as 0.
this is more like an issue from sensor side, you should review your sensor configuration.

system · November 20, 2023, 4:42am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.