【Jetson AGX ORIN】Camera fails to capture images intermittently

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Hi

I am testing the stability of GMSL cameras on Jetson AGX Orin 64G, and the camera sensor used is ISX031.

  • Hardware Information:

    • max96712 2*4lane(PortA/PortB) D-PHY
    • 4*isx031(1920**1536)

  • Software Info: Jetpack 6.2 [L4T 36.4.3]

    • mipi clock = 1100000000(Other parameters have also been configured)

  • demo process:

    1. Control the power-off and power-on of the GMSL camera POC, as well as the power-off and power-on of the des.
    2. Initialize the camera link by sending register configurations via ioctl, including sensors and serdes.
    3. Capture images via v4l2

  • Issue Description: I repeatedly restart the demo to verify camera stability. Occasionally, four cameras fail to capture images. Without performing any other operations, re-capturing images using the following commands will restore normal functionality.

    • v4l2-ctl --set-fmt-video=width=1920,height=1536 --stream-mmap --stream-count=10000000000 -d /dev/video0
    • v4l2-ctl --set-fmt-video=width=1920,height=1536 --stream-mmap --stream-count=10000000000 -d /dev/video1
    • v4l2-ctl --set-fmt-video=width=1920,height=1536 --stream-mmap --stream-count=10000000000 -d /dev/video2
    • v4l2-ctl --set-fmt-video=width=1920,height=1536 --stream-mmap --stream-count=10000000000 -d /dev/video3

  • Below are the logs and screenshots obtained from the tests.
    The output_80..log is the log generated when an error occurred during the 80th launch of the demo.

  • An error may occur once after running the demo hundreds of times, or sometimes just dozens of times. The occurrence probability is highly unstable.

  • echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace

     camera_demo-15123   [008] .......  1733.954916: tegra_channel_open: vi-output, imx185 2-001b
     camera_demo-15123   [008] .......  1733.955113: tegra_channel_close: vi-output, imx185 2-001b
     camera_demo-15123   [008] .......  1734.227359: tegra_channel_open: vi-output, imx185 2-001c
     camera_demo-15123   [008] .......  1734.227602: tegra_channel_close: vi-output, imx185 2-001c
     camera_demo-15123   [008] .......  1734.496119: tegra_channel_open: vi-output, imx185 2-001d
     camera_demo-15123   [008] .......  1734.496219: tegra_channel_close: vi-output, imx185 2-001d
     camera_demo-15123   [008] .......  1734.769085: tegra_channel_open: vi-output, imx185 2-001e
     camera_demo-15123   [008] .......  1734.769301: tegra_channel_close: vi-output, imx185 2-001e
     camera_demo-15123   [008] .......  1734.770686: tegra_channel_open: vi-output, imx185 2-001b
     camera_demo-15123   [008] .......  1734.782347: tegra_channel_open: vi-output, imx185 2-001c
     camera_demo-15123   [008] .......  1734.792634: tegra_channel_open: vi-output, imx185 2-001d
     camera_demo-15123   [008] .......  1734.803037: tegra_channel_open: vi-output, imx185 2-001e
     camera_demo-15166   [010] .......  1734.843008: tegra_channel_set_power: imx185 2-001c : 0x1
     camera_demo-15165   [011] .......  1734.843009: tegra_channel_set_power: imx185 2-001b : 0x1
     camera_demo-15168   [008] .......  1734.843011: tegra_channel_set_power: imx185 2-001e : 0x1
     camera_demo-15167   [009] .......  1734.843014: tegra_channel_set_power: imx185 2-001d : 0x1
     camera_demo-15166   [010] .......  1734.843019: camera_common_s_power: status : 0x1
     camera_demo-15165   [011] .......  1734.843020: camera_common_s_power: status : 0x1
     camera_demo-15168   [008] .......  1734.843020: camera_common_s_power: status : 0x1
     camera_demo-15167   [009] .......  1734.843023: camera_common_s_power: status : 0x1
     camera_demo-15166   [010] .......  1734.843905: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-15166   [010] .......  1734.843908: csi_s_power: enable : 0x1
     camera_demo-15165   [011] .......  1734.844369: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-15165   [011] .......  1734.844371: csi_s_power: enable : 0x1
     camera_demo-15168   [008] .......  1734.844504: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-15168   [008] .......  1734.844505: csi_s_power: enable : 0x1
     camera_demo-15167   [009] .......  1734.844634: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-15167   [009] .......  1734.844635: csi_s_power: enable : 0x1
     camera_demo-15168   [008] .......  1734.846157: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
     camera_demo-15166   [010] .......  1734.876660: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-15169   [007] .......  1734.876883: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:17466 pid:15169 tid:15169
 vi-output, imx1-15169   [007] .......  1734.876900: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:17466 pid:15169 tid:15169
     camera_demo-15168   [008] .......  1734.877123: tegra_channel_set_stream: enable : 0x1
     camera_demo-15168   [008] .......  1734.880879: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-15168   [008] .......  1734.880881: csi_s_stream: enable : 0x1
     camera_demo-15167   [009] .......  1734.897677: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-15171   [001] .......  1734.897816: vi_task_submit: class_id:48 ch:1 syncpt_id:40 syncpt_thresh:17460 pid:15171 tid:15171
 vi-output, imx1-15171   [001] .......  1734.897825: vi_task_submit: class_id:48 ch:1 syncpt_id:40 syncpt_thresh:17460 pid:15171 tid:15171
     camera_demo-15166   [010] .......  1734.897915: tegra_channel_set_stream: enable : 0x1
     camera_demo-15166   [010] .......  1734.902365: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-15166   [010] .......  1734.902367: csi_s_stream: enable : 0x1
     camera_demo-15165   [008] .......  1734.918418: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-15173   [006] .......  1734.918543: vi_task_submit: class_id:48 ch:2 syncpt_id:43 syncpt_thresh:17468 pid:15173 tid:15173
 vi-output, imx1-15173   [006] .......  1734.918556: vi_task_submit: class_id:48 ch:2 syncpt_id:43 syncpt_thresh:17468 pid:15173 tid:15173
     camera_demo-15167   [009] .......  1734.918570: tegra_channel_set_stream: enable : 0x1
     camera_demo-15167   [009] .......  1734.921449: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-15167   [009] .......  1734.921451: csi_s_stream: enable : 0x1
     camera_demo-15168   [008] .......  1734.921806: tegra_channel_set_stream: imx185 2-001e : 0x1
     camera_demo-15165   [008] .......  1734.939190: tegra_channel_set_stream: enable : 0x1
 vi-output, imx1-15175   [004] .......  1734.939253: vi_task_submit: class_id:48 ch:3 syncpt_id:46 syncpt_thresh:17420 pid:15175 tid:15175
 vi-output, imx1-15175   [004] .......  1734.939264: vi_task_submit: class_id:48 ch:3 syncpt_id:46 syncpt_thresh:17420 pid:15175 tid:15175
     camera_demo-15165   [008] .......  1734.940809: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-15165   [008] .......  1734.940810: csi_s_stream: enable : 0x1

  • Jan  1 08:29:18 ubuntu kernel: [ 1734.846144] [VI_SETUP] Acquired operation lock for channel 3, vi_port 0
Jan  1 08:29:18 ubuntu kernel: [ 1734.846214] [IVC_REGISTER] Registering control callback: cb_func=00000000bc416144, priv_context=000000001d23a140
Jan  1 08:29:18 ubuntu kernel: [ 1734.855042] [IVC_REGISTER] Control callback registered successfully: trans_id=140, cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.876595] [IVC_NOTIFY] Channel ID notification: chan_id=0, trans_id=140
Jan  1 08:29:18 ubuntu kernel: [ 1734.876602] [IVC_NOTIFY] Before validation: trans_id=140 cb_func=00000000bc416144, chan_id=0 cb_func=0000000000000000
Jan  1 08:29:18 ubuntu kernel: [ 1734.876607] [IVC_NOTIFY] Callback mapping: trans_id=140 -> chan_id=0, cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.876611] [IVC_NOTIFY] After mapping: trans_id=140 cb_func=0000000000000000, chan_id=0 cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.876620] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 3, vi_port 0
Jan  1 08:29:18 ubuntu kernel: [ 1734.876646] [VI_SETUP] Acquired operation lock for channel 1, vi_port 0
Jan  1 08:29:18 ubuntu kernel: [ 1734.876714] [IVC_REGISTER] Registering control callback: cb_func=00000000bc416144, priv_context=000000004e08bf6a
Jan  1 08:29:18 ubuntu kernel: [ 1734.876726] [IVC_REGISTER] Control callback registered successfully: trans_id=141, cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.897605] [IVC_NOTIFY] Channel ID notification: chan_id=1, trans_id=141
Jan  1 08:29:18 ubuntu kernel: [ 1734.897615] [IVC_NOTIFY] Before validation: trans_id=141 cb_func=00000000bc416144, chan_id=1 cb_func=0000000000000000
Jan  1 08:29:18 ubuntu kernel: [ 1734.897625] [IVC_NOTIFY] Callback mapping: trans_id=141 -> chan_id=1, cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.897632] [IVC_NOTIFY] After mapping: trans_id=141 cb_func=0000000000000000, chan_id=1 cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.897646] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 1, vi_port 0
Jan  1 08:29:18 ubuntu kernel: [ 1734.897667] [VI_SETUP] Acquired operation lock for channel 2, vi_port 0
Jan  1 08:29:18 ubuntu kernel: [ 1734.897713] [IVC_REGISTER] Registering control callback: cb_func=00000000bc416144, priv_context=00000000c11bd83f
Jan  1 08:29:18 ubuntu kernel: [ 1734.897722] [IVC_REGISTER] Control callback registered successfully: trans_id=142, cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.918372] [IVC_NOTIFY] Channel ID notification: chan_id=2, trans_id=142
Jan  1 08:29:18 ubuntu kernel: [ 1734.918378] [IVC_NOTIFY] Before validation: trans_id=142 cb_func=00000000bc416144, chan_id=2 cb_func=0000000000000000
Jan  1 08:29:18 ubuntu kernel: [ 1734.918383] [IVC_NOTIFY] Callback mapping: trans_id=142 -> chan_id=2, cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.918387] [IVC_NOTIFY] After mapping: trans_id=142 cb_func=0000000000000000, chan_id=2 cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.918394] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 2, vi_port 0
Jan  1 08:29:18 ubuntu kernel: [ 1734.918413] [VI_SETUP] Acquired operation lock for channel 0, vi_port 0
Jan  1 08:29:18 ubuntu kernel: [ 1734.918440] [IVC_REGISTER] Registering control callback: cb_func=00000000bc416144, priv_context=00000000a69a46a5
Jan  1 08:29:18 ubuntu kernel: [ 1734.918444] [IVC_REGISTER] Control callback registered successfully: trans_id=143, cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.939044] [IVC_NOTIFY] Channel ID notification: chan_id=3, trans_id=143
Jan  1 08:29:18 ubuntu kernel: [ 1734.939051] [IVC_NOTIFY] Before validation: trans_id=143 cb_func=00000000bc416144, chan_id=3 cb_func=0000000000000000
Jan  1 08:29:18 ubuntu kernel: [ 1734.939056] [IVC_NOTIFY] Callback mapping: trans_id=143 -> chan_id=3, cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.939059] [IVC_NOTIFY] After mapping: trans_id=143 cb_func=0000000000000000, chan_id=3 cb_func=00000000bc416144
Jan  1 08:29:18 ubuntu kernel: [ 1734.939066] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 0, vi_port 0
Jan  1 08:29:21 ubuntu kernel: [ 1737.414352] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Jan  1 08:29:21 ubuntu kernel: [ 1737.414373] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Jan  1 08:29:21 ubuntu kernel: [ 1737.414377] [VI_RELEASE] Acquired operation lock for channel 1, reset_flags 0x1
Jan  1 08:29:21 ubuntu kernel: [ 1737.414388] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Jan  1 08:29:21 ubuntu kernel: [ 1737.414404] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Jan  1 08:29:21 ubuntu kernel: [ 1737.435237] [IVC_UNREGISTER] Unregistering control callback: id=1
Jan  1 08:29:21 ubuntu kernel: [ 1737.435241] [IVC_UNREGISTER] Before unregister: id=1, cb_func=00000000bc416144
Jan  1 08:29:21 ubuntu kernel: [ 1737.435244] [IVC_UNREGISTER] Control callback unregistered: id=1
Jan  1 08:29:21 ubuntu kernel: [ 1737.435256] [VI_RELEASE] Release completed, releasing operation lock for channel 65535
Jan  1 08:29:21 ubuntu kernel: [ 1737.435443] [VI_SETUP] Acquired operation lock for channel 1, vi_port 0
Jan  1 08:29:21 ubuntu kernel: [ 1737.435483] [IVC_REGISTER] Registering control callback: cb_func=00000000bc416144, priv_context=000000009cf7fae6
Jan  1 08:29:21 ubuntu kernel: [ 1737.435487] [IVC_REGISTER] Control callback registered successfully: trans_id=144, cb_func=00000000bc416144
Jan  1 08:29:21 ubuntu kernel: [ 1737.458035] [IVC_NOTIFY] Channel ID notification: chan_id=1, trans_id=144
Jan  1 08:29:21 ubuntu kernel: [ 1737.458041] [IVC_NOTIFY] Before validation: trans_id=144 cb_func=00000000bc416144, chan_id=1 cb_func=0000000000000000
Jan  1 08:29:21 ubuntu kernel: [ 1737.458046] [IVC_NOTIFY] Callback mapping: trans_id=144 -> chan_id=1, cb_func=00000000bc416144
Jan  1 08:29:21 ubuntu kernel: [ 1737.458050] [IVC_NOTIFY] After mapping: trans_id=144 cb_func=0000000000000000, chan_id=1 cb_func=00000000bc416144
Jan  1 08:29:21 ubuntu kernel: [ 1737.458057] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 1, vi_port 0
Jan  1 08:29:21 ubuntu kernel: [ 1737.458065] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
Jan  1 08:29:21 ubuntu kernel: [ 1737.458070] [VI_RELEASE] Acquired operation lock for channel 0, reset_flags 0x1

  • camera_log.zip (1.8 MB)

    image
    image1164×1102 591 KB

2

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, it appears that you are experiencing issues with the stability of GMSL cameras on Jetson AGX Orin 64G, specifically with the ISX031 camera sensor. The issue is intermittent, and the camera fails to capture images occasionally, but recapturing images using the v4l2ctl command restores normal functionality.

To address this issue, let’s break down the key points from the provided context:

  1. Hardware Information: The setup consists of a Jetson AGX Orin 64G module, a max96712 serializer, and four ISX031 camera sensors connected to the serializer via DPHY lanes.
  2. Software Information: The system is running Jetpack 6.2, L4T 36.4.3, and the mipi clock is set to 1100000000 Hz.
  3. Demo Process: The demo process involves controlling the power-off and power-on of the GMSL camera POC, initializing the camera link by sending register configurations via ioctl, and capturing images via v4l2.
  4. Issue Description: The camera occasionally fails to capture images, but recapturing images using the v4l2ctl command restores normal functionality.

To troubleshoot this issue, let’s consider the following potential causes:

  1. Hardware Issues: It’s possible that there are hardware issues with the camera sensors, serializer, or the Jetson module itself. However, the fact that recapturing images using the v4l2ctl command restores normal functionality suggests that the hardware is likely not the primary cause of the issue.
  2. Software Issues: There might be software issues related to the camera driver, v4l2, or the ioctl commands used to configure the camera link. The fact that the issue is intermittent and the camera fails to capture images occasionally suggests that there might be a software-related issue.
  3. Configuration Issues: It’s possible that there are configuration issues related to the camera link, such as incorrect register settings or mipi clock configuration.

To further troubleshoot this issue, I would recommend the following steps:

  1. Verify Camera Link Configuration: Verify that the camera link configuration is correct, including the register settings and mipi clock configuration.
  2. Check Camera Driver: Check the camera driver version and ensure that it is compatible with the Jetson AGX Orin 64G module and the ISX031 camera sensor.
  3. Run v4l2 Compliance Test: Run the v4l2 compliance test to ensure that the camera driver is functioning correctly.
  4. Monitor System Logs: Monitor the system logs to see if there are any error messages related to the camera or v4l2.
  5. Test with Different Camera Sensors: Test the setup with different camera sensors to see if the issue is specific to the ISX031 sensor.

In comparison to other issues mentioned in the provided context, this issue appears to be specific to the GMSL camera setup on Jetson AGX Orin 64G with the ISX031 camera sensor. However, some of the other issues mentioned, such as the “V4l2 DMA extension support for opengl (EGL_LINUX_DMA_BUF_EXT) is not working” issue, might be related to the camera driver or v4l2 configuration.

In summary, the issue appears to be related to the camera link configuration or software issues, and further troubleshooting is needed to identify the root cause of the issue.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Hi，

These responses do not seem to offer effective solutions. This issue occurs randomly, and every time it happens, simply closing the image capture program and restarting the capture process allows image data to be retrieved normally.

5

hello wangyu3,

let’s have a quick try to disable RCE suspend (rtcpu auto-suspend) for testing.
# echo -1 > /sys/devices/platform/*rtcpu/power/autosuspend_delay_ms
# echo on > /sys/devices/platform/*rtcpu/power/control

6

Hi

Do we still need to capture traces after configuration?

7

Hi

我进行了以下操作

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace
echo -1 > /sys/devices/platform/*rtcpu/power/autosuspend_delay_ms
echo on > /sys/devices/platform/*rtcpu/power/control

在测试第530次复现了问题

image
image1556×1114 162 KB

image
image1920×1034 1.26 MB

   camera_demo-1038310 [009] ....... 299041.509106: tegra_channel_open: vi-output, imx185 2-001c
     camera_demo-1038310 [009] ....... 299041.519953: tegra_channel_open: vi-output, imx185 2-001d
     camera_demo-1038310 [009] ....... 299041.530199: tegra_channel_open: vi-output, imx185 2-001e
     camera_demo-1038357 [009] ....... 299041.568138: tegra_channel_set_power: imx185 2-001b : 0x1
     camera_demo-1038359 [008] ....... 299041.568140: tegra_channel_set_power: imx185 2-001d : 0x1
     camera_demo-1038360 [010] ....... 299041.568141: tegra_channel_set_power: imx185 2-001e : 0x1
     camera_demo-1038357 [009] ....... 299041.568196: camera_common_s_power: status : 0x1
     camera_demo-1038360 [010] ....... 299041.568197: camera_common_s_power: status : 0x1
     camera_demo-1038359 [008] ....... 299041.568197: camera_common_s_power: status : 0x1
     camera_demo-1038358 [011] ....... 299041.568277: tegra_channel_set_power: imx185 2-001c : 0x1
     camera_demo-1038358 [011] ....... 299041.568284: camera_common_s_power: status : 0x1
     camera_demo-1038359 [008] ....... 299041.568818: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-1038359 [008] ....... 299041.568820: csi_s_power: enable : 0x1
     camera_demo-1038357 [009] ....... 299041.568985: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-1038357 [009] ....... 299041.568986: csi_s_power: enable : 0x1
     camera_demo-1038360 [010] ....... 299041.569130: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-1038360 [010] ....... 299041.569131: csi_s_power: enable : 0x1
     camera_demo-1038358 [011] ....... 299041.569550: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-1038358 [011] ....... 299041.569552: csi_s_power: enable : 0x1
     camera_demo-1038360 [010] ....... 299041.570033: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038361 [001] ....... 299041.590965: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:185558 pid:1038361 tid:1038361
 vi-output, imx1-1038361 [001] ....... 299041.590982: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:185558 pid:1038361 tid:1038361
     camera_demo-1038360 [011] ....... 299041.591031: tegra_channel_set_stream: enable : 0x1
     camera_demo-1038359 [009] ....... 299041.591046: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
     camera_demo-1038360 [011] ....... 299041.595925: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-1038360 [011] ....... 299041.595929: csi_s_stream: enable : 0x1
     camera_demo-1038358 [011] ....... 299041.611825: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038363 [000] ....... 299041.611946: vi_task_submit: class_id:48 ch:1 syncpt_id:40 syncpt_thresh:185561 pid:1038363 tid:1038363
 vi-output, imx1-1038363 [000] ....... 299041.611960: vi_task_submit: class_id:48 ch:1 syncpt_id:40 syncpt_thresh:185561 pid:1038363 tid:1038363
     camera_demo-1038359 [009] ....... 299041.611973: tegra_channel_set_stream: enable : 0x1
     camera_demo-1038359 [009] ....... 299041.617399: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-1038359 [009] ....... 299041.617400: csi_s_stream: enable : 0x1
     camera_demo-1038357 [009] ....... 299041.632521: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038365 [006] ....... 299041.632626: vi_task_submit: class_id:48 ch:2 syncpt_id:43 syncpt_thresh:185595 pid:1038365 tid:1038365
     camera_demo-1038358 [011] ....... 299041.632656: tegra_channel_set_stream: enable : 0x1
 vi-output, imx1-1038365 [006] ....... 299041.632658: vi_task_submit: class_id:48 ch:2 syncpt_id:43 syncpt_thresh:185595 pid:1038365 tid:1038365
     camera_demo-1038358 [011] ....... 299041.636286: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-1038358 [011] ....... 299041.636287: csi_s_stream: enable : 0x1
     camera_demo-1038360 [011] ....... 299041.637289: tegra_channel_set_stream: imx185 2-001e : 0x1
 vi-output, imx1-1038367 [004] ....... 299041.653295: vi_task_submit: class_id:48 ch:3 syncpt_id:46 syncpt_thresh:185200 pid:1038367 tid:1038367
 vi-output, imx1-1038367 [004] ....... 299041.653306: vi_task_submit: class_id:48 ch:3 syncpt_id:46 syncpt_thresh:185200 pid:1038367 tid:1038367
     camera_demo-1038357 [009] ....... 299041.653333: tegra_channel_set_stream: enable : 0x1
     camera_demo-1038357 [009] ....... 299041.656744: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
     camera_demo-1038357 [009] ....... 299041.656746: csi_s_stream: enable : 0x1
     camera_demo-1038359 [009] ....... 299041.658523: tegra_channel_set_stream: imx185 2-001d : 0x1
     camera_demo-1038358 [011] ....... 299041.677414: tegra_channel_set_stream: imx185 2-001c : 0x1
     camera_demo-1038357 [009] ....... 299041.697672: tegra_channel_set_stream: imx185 2-001b : 0x1
 vi-output, imx1-1038362 [007] ....... 299044.307174: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038361 [001] ....... 299044.327832: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:185558 pid:1038361 tid:1038361
 vi-output, imx1-1038361 [001] ....... 299044.327845: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:185558 pid:1038361 tid:1038361
 vi-output, imx1-1038364 [005] ....... 299044.390713: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038366 [007] ....... 299044.411521: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038363 [000] ....... 299044.411522: vi_task_submit: class_id:48 ch:1 syncpt_id:40 syncpt_thresh:185561 pid:1038363 tid:1038363
 vi-output, imx1-1038363 [000] ....... 299044.411535: vi_task_submit: class_id:48 ch:1 syncpt_id:40 syncpt_thresh:185561 pid:1038363 tid:1038363
 vi-output, imx1-1038365 [004] ....... 299044.432143: vi_task_submit: class_id:48 ch:2 syncpt_id:43 syncpt_thresh:185595 pid:1038365 tid:1038365
 vi-output, imx1-1038365 [004] ....... 299044.432151: vi_task_submit: class_id:48 ch:2 syncpt_id:43 syncpt_thresh:185595 pid:1038365 tid:1038365
 vi-output, imx1-1038368 [005] ....... 299044.432281: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038367 [008] ....... 299044.453040: vi_task_submit: class_id:48 ch:3 syncpt_id:46 syncpt_thresh:185200 pid:1038367 tid:1038367
 vi-output, imx1-1038367 [008] ....... 299044.453051: vi_task_submit: class_id:48 ch:3 syncpt_id:46 syncpt_thresh:185200 pid:1038367 tid:1038367
 vi-output, imx1-1038362 [007] ....... 299046.867766: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038361 [001] ....... 299046.890197: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:185558 pid:1038361 tid:1038361
 vi-output, imx1-1038361 [001] ....... 299046.890212: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:185558 pid:1038361 tid:1038361
 vi-output, imx1-1038368 [007] ....... 299047.166190: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038364 [005] ....... 299047.186825: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038367 [010] ....... 299047.186825: vi_task_submit: class_id:48 ch:1 syncpt_id:40 syncpt_thresh:185561 pid:1038367 tid:1038367
 vi-output, imx1-1038367 [010] ....... 299047.186836: vi_task_submit: class_id:48 ch:1 syncpt_id:40 syncpt_thresh:185561 pid:1038367 tid:1038367
 vi-output, imx1-1038366 [005] ....... 299047.207462: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038363 [001] ....... 299047.207462: vi_task_submit: class_id:48 ch:2 syncpt_id:43 syncpt_thresh:185595 pid:1038363 tid:1038363
 vi-output, imx1-1038363 [001] ....... 299047.207472: vi_task_submit: class_id:48 ch:2 syncpt_id:43 syncpt_thresh:185595 pid:1038363 tid:1038363
 vi-output, imx1-1038365 [004] ....... 299047.228081: vi_task_submit: class_id:48 ch:3 syncpt_id:46 syncpt_thresh:185200 pid:1038365 tid:1038365
 vi-output, imx1-1038365 [004] ....... 299047.228088: vi_task_submit: class_id:48 ch:3 syncpt_id:46 syncpt_thresh:185200 pid:1038365 tid:1038365
 vi-output, imx1-1038362 [007] ....... 299049.427862: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1536 fmt 10
 vi-output, imx1-1038361 [001] ....... 299049.448593: vi_task_submit: class_id:48 ch:0 syncpt_id:37 syncpt_thresh:185558 pid:1038361 tid:1038361

trace.txt (89.7 MB)

Jun 10 14:42:15 ubuntu kernel: [299041.570068] [IVC_REGISTER] Registering control callback: cb_func=0000000069d07b7e, priv_context=00000000d29381ea
Jun 10 14:42:15 ubuntu kernel: [299041.570076] [IVC_REGISTER] Control callback registered successfully: trans_id=104, cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.590833] [IVC_NOTIFY] Channel ID notification: chan_id=0, trans_id=104
Jun 10 14:42:15 ubuntu kernel: [299041.590839] [IVC_NOTIFY] Before validation: trans_id=104 cb_func=0000000069d07b7e, chan_id=0 cb_func=0000000000000000
Jun 10 14:42:15 ubuntu kernel: [299041.590843] [IVC_NOTIFY] Callback mapping: trans_id=104 -> chan_id=0, cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.590845] [IVC_NOTIFY] After mapping: trans_id=104 cb_func=0000000000000000, chan_id=0 cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.590850] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 3, vi_port 0
Jun 10 14:42:15 ubuntu kernel: [299041.591034] [VI_SETUP] Acquired operation lock for channel 2, vi_port 0
Jun 10 14:42:15 ubuntu kernel: [299041.591107] [IVC_REGISTER] Registering control callback: cb_func=0000000069d07b7e, priv_context=000000008b93d80f
Jun 10 14:42:15 ubuntu kernel: [299041.591117] [IVC_REGISTER] Control callback registered successfully: trans_id=105, cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.611778] [IVC_NOTIFY] Channel ID notification: chan_id=1, trans_id=105
Jun 10 14:42:15 ubuntu kernel: [299041.611785] [IVC_NOTIFY] Before validation: trans_id=105 cb_func=0000000069d07b7e, chan_id=1 cb_func=0000000000000000
Jun 10 14:42:15 ubuntu kernel: [299041.611791] [IVC_NOTIFY] Callback mapping: trans_id=105 -> chan_id=1, cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.611795] [IVC_NOTIFY] After mapping: trans_id=105 cb_func=0000000000000000, chan_id=1 cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.611803] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 2, vi_port 0
Jun 10 14:42:15 ubuntu kernel: [299041.611820] [VI_SETUP] Acquired operation lock for channel 1, vi_port 0
Jun 10 14:42:15 ubuntu kernel: [299041.611867] [IVC_REGISTER] Registering control callback: cb_func=0000000069d07b7e, priv_context=0000000065b2d6b5
Jun 10 14:42:15 ubuntu kernel: [299041.611872] [IVC_REGISTER] Control callback registered successfully: trans_id=106, cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.632476] [IVC_NOTIFY] Channel ID notification: chan_id=2, trans_id=106
Jun 10 14:42:15 ubuntu kernel: [299041.632482] [IVC_NOTIFY] Before validation: trans_id=106 cb_func=0000000069d07b7e, chan_id=2 cb_func=0000000000000000
Jun 10 14:42:15 ubuntu kernel: [299041.632486] [IVC_NOTIFY] Callback mapping: trans_id=106 -> chan_id=2, cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.632489] [IVC_NOTIFY] After mapping: trans_id=106 cb_func=0000000000000000, chan_id=2 cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.632495] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 1, vi_port 0
Jun 10 14:42:15 ubuntu kernel: [299041.632515] [VI_SETUP] Acquired operation lock for channel 0, vi_port 0
Jun 10 14:42:15 ubuntu kernel: [299041.632551] [IVC_REGISTER] Registering control callback: cb_func=0000000069d07b7e, priv_context=00000000ace4fdd2
Jun 10 14:42:15 ubuntu kernel: [299041.632557] [IVC_REGISTER] Control callback registered successfully: trans_id=107, cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.653178] [IVC_NOTIFY] Channel ID notification: chan_id=3, trans_id=107
Jun 10 14:42:15 ubuntu kernel: [299041.653188] [IVC_NOTIFY] Before validation: trans_id=107 cb_func=0000000069d07b7e, chan_id=3 cb_func=0000000000000000
Jun 10 14:42:15 ubuntu kernel: [299041.653198] [IVC_NOTIFY] Callback mapping: trans_id=107 -> chan_id=3, cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.653205] [IVC_NOTIFY] After mapping: trans_id=107 cb_func=0000000000000000, chan_id=3 cb_func=0000000069d07b7e
Jun 10 14:42:15 ubuntu kernel: [299041.653214] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 0, vi_port 0
Jun 10 14:42:18 ubuntu kernel: [299044.285915] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Jun 10 14:42:18 ubuntu kernel: [299044.285929] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Jun 10 14:42:18 ubuntu kernel: [299044.285942] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Jun 10 14:42:18 ubuntu kernel: [299044.285950] [VI_RELEASE] Acquired operation lock for channel 0, reset_flags 0x1
Jun 10 14:42:18 ubuntu kernel: [299044.285957] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Jun 10 14:42:18 ubuntu kernel: [299044.286002] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Jun 10 14:42:18 ubuntu kernel: [299044.286019] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Jun 10 14:42:18 ubuntu kernel: [299044.286108] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Jun 10 14:42:18 ubuntu kernel: [299044.286129] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Jun 10 14:42:18 ubuntu kernel: [299044.306981] [IVC_UNREGISTER] Unregistering control callback: id=0
Jun 10 14:42:18 ubuntu kernel: [299044.306986] [IVC_UNREGISTER] Before unregister: id=0, cb_func=0000000069d07b7e
Jun 10 14:42:18 ubuntu kernel: [299044.306990] [IVC_UNREGISTER] Control callback unregistered: id=0
Jun 10 14:42:18 ubuntu kernel: [299044.307005] [VI_RELEASE] Release completed, releasing operation lock for channel 65535
Jun 10 14:42:18 ubuntu kernel: [299044.307166] [VI_SETUP] Acquired operation lock for channel 3, vi_port 0
Jun 10 14:42:18 ubuntu kernel: [299044.307211] [IVC_REGISTER] Registering control callback: cb_func=0000000069d07b7e, priv_context=00000000a04a6777
Jun 10 14:42:18 ubuntu kernel: [299044.307215] [IVC_REGISTER] Control callback registered successfully: trans_id=108, cb_func=0000000069d07b7e
Jun 10 14:42:18 ubuntu kernel: [299044.327807] [IVC_NOTIFY] Channel ID notification: chan_id=0, trans_id=108
Jun 10 14:42:18 ubuntu kernel: [299044.327811] [IVC_NOTIFY] Before validation: trans_id=108 cb_func=0000000069d07b7e, chan_id=0 cb_func=0000000000000000
Jun 10 14:42:18 ubuntu kernel: [299044.327814] [IVC_NOTIFY] Callback mapping: trans_id=108 -> chan_id=0, cb_func=0000000069d07b7e
Jun 10 14:42:18 ubuntu kernel: [299044.327816] [IVC_NOTIFY] After mapping: trans_id=108 cb_func=0000000000000000, chan_id=0 cb_func=0000000069d07b7e
Jun 10 14:42:18 ubuntu kernel: [299044.327820] [VI_SETUP] Setup completed successfully, releasing operation lock for channel 3, vi_port 0
Jun 10 14:42:18 ubuntu kernel: [299044.327826] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
Jun 10 14:42:18 ubuntu kernel: [299044.328022] [VI_RELEASE] Acquired operation lock for channel 1, reset_flags 0x1
Jun 10 14:42:18 ubuntu kernel: [299044.348710] [IVC_UNREGISTER] Unregistering control callback: id=1
Jun 10 14:42:18 ubuntu kernel: [299044.348712] [IVC_UNREGISTER] Before unregister: id=1, cb_func=0000000069d07b7e
Jun 10 14:42:18 ubuntu kernel: [299044.348713] [IVC_UNREGISTER] Control callback unregistered: id=1
Jun 10 14:42:18 ubuntu kernel: [299044.348720] [VI_RELEASE] Release completed, releasing operation lock for channel 65535
Jun 10 14:42:18 ubuntu kernel: [299044.348870] [VI_RELEASE] Acquired operation lock for channel 2, reset_flags 0x1
Jun 10 14:42:18 ubuntu kernel: [299044.369751] [IVC_UNREGISTER] Unregistering control callback: id=2
Jun 10 14:42:18 ubuntu kernel: [299044.369753] [IVC_UNREGISTER] Before unregister: id=2, cb_func=0000000069d07b7e
Jun 10 14:42:18 ubuntu kernel: [299044.369754] [IVC_UNREGISTER] Control callback unregistered: id=2
Jun 10 14:42:18 ubuntu kernel: [299044.369761] [VI_RELEASE] Release completed, releasing operation lock for channel 65535
Jun 10 14:42:18 ubuntu kernel: [299044.369770] [VI_RELEASE] Acquired operation lock for channel 3, reset_flags 0x1
Jun 10 14:42:18 ubuntu kernel: [299044.390687] [IVC_UNREGISTER] Unregistering control callback: id=3
Jun 10 14:42:18 ubuntu kernel: [299044.390689] [IVC_UNREGISTER] Before unregister: id=3, cb_func=0000000069d07b7e
Jun 10 14:42:18 ubuntu kernel: [299044.390691] [IVC_UNREGISTER] Control callback unregistered: id=3
Jun 10 14:42:18 ubuntu kernel: [299044.390696] [VI_RELEASE] Release completed, releasing operation lock for channel 65535
Jun 10 14:42:18 ubuntu kernel: [299044.390704] [VI_SETUP] Acquired operation lock for channel 2, vi_port 0
8

hello wangyu3,

as you can see.. there’s error recover mechanism to restore the stream.

please see-also Topic 337341, you may give it a try to apply the kernel patch, which has semaphore added to avoid multi-cam race condition.

9

Hi

Is this patch officially merged into which JP version?

10

hello wangyu3,

that is included in Jetpack-6.2.2/l4t-r36.5.

11

Hi,

The problem can still be reproduced after I merged the code.

Are there any other troubleshooting directions?

image
image2044×1321 506 KB

12

Hi，

Do we need to merge the provided Patch above into JETSON THOR?

13

hello wangyu3,

may I double check which Jetpack release version you’re working with.
besides.. let’s start a new forum thread if you’re moving to Thor platform.

14

Hi,
I found that the corresponding patches have not been merged into both JP7.0 and JP7.1.

Should this be merged into Thor?

16

hello wangyu3,

please check the latest JetPack 7.2 release version,
the patch to add semaphore to avoid multi-cam race condition has included.

17

Hi，

Okay, noted. I will internally merge the corresponding patches into JP7.0 and JP7.1.

I have also been conducting relevant camera tests recently, and I suspect the issue is related to VI. I added corresponding registers on the serializer and deserializer to retrieve real-time status, yet no register abnormalities were detected when the problem occurred. What troubleshooting methods are available for the VI module?

18

Hi，

Currently, my testing is still on ORIN JP6.2

19

hello wangyu3,

is this related to multi-cam race condition? can you try reduce the number of cameras?
if that’s possible, please try reproduce this with.. Jetpack-6.2.2/l4t-r36.5.