Jetson orin shutdown caused by multi camera application

gregoire.zeisser2 · March 25, 2025, 12:24pm

Hello,

I am using a package Jetson Orin, with Jetpack 5.1.2.

When starting a multi camera application, it sometimes causes the orin to reboot, with “channel context at x is busy” and a call trace showing a problem in tegra_capture_ivc_notify_chan_id.
I am running a simple gstreamer pipeline for acquisition of 6 camera (30 fps, 1920*1280) and mux/demux them :

gst-launch-1.0 \
    v4l2src device=/dev/video0 ! videoconvert ! 'video/x-raw,format=NV12,width=1920,height=1280' ! queue ! mix.sink_0 \
    v4l2src device=/dev/video1 ! videoconvert ! 'video/x-raw,format=NV12,width=1920,height=1280' ! queue ! mix.sink_1 \
    v4l2src device=/dev/video2 ! videoconvert ! 'video/x-raw,format=NV12,width=1920,height=1280' ! queue ! mix.sink_2 \
    v4l2src device=/dev/video3 ! videoconvert ! 'video/x-raw,format=NV12,width=1920,height=1280' ! queue ! mix.sink_3 \
    v4l2src device=/dev/video4 ! videoconvert ! 'video/x-raw,format=NV12,width=1920,height=1280' ! queue ! mix.sink_4 \
    v4l2src device=/dev/video5 ! videoconvert ! 'video/x-raw,format=NV12,width=1920,height=1280' ! queue ! mix.sink_5 \
    compositor name=mix background=black \
    ! queue \
    ! tee name=t \
    t. ! queue ! fakesink \
    t. ! queue ! fakesink \
    t. ! queue ! fakesink \
    t. ! queue ! fakesink \
    t. ! queue ! fakesink \
    t. ! queue ! fakesink

extract from the kern.log :

Mar 21 12:23:56 localhost kernel: [  257.065956] ------------[ cut here ]------------
Mar 21 12:23:56 localhost kernel: [  257.076161] (NULL device *): vi_capture_control_message: NULL VI channel received
Mar 21 12:23:56 localhost kernel: [  257.079958] tier4_isx021 42-001c: [tier4_isx021_start_one_streaming] : Enabled Auto Exposure.
Mar 21 12:23:56 localhost kernel: [  257.079960] tier4_isx021 42-001c: [tier4_isx021_start_one_streaming] : fsync-mode in DTB = 0
Mar 21 12:23:56 localhost kernel: [  257.080776] channel context at 1 is busy
Mar 21 12:23:56 localhost kernel: [  257.080802] WARNING: CPU: 1 PID: 4162 at /home/ryomunakata/dev/src/linux-cti-anvil/sources/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:176 tegra_capture_ivc_notify_chan_id+0x18c/0x1b0
Mar 21 12:23:56 localhost kernel: [  257.088388] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 5 vc- 0
Mar 21 12:23:56 localhost kernel: [  257.105798] Modules linked in: nvidia_modeset(OE) xt_multiport(E) xt_tcpudp(E)
Mar 21 12:23:56 localhost kernel: [  257.110321] tier4_isx021 42-001c: [tier4_isx021_start_one_streaming] : Prameter[enable_distortion_correction] = 1 .
Mar 21 12:23:56 localhost kernel: [  257.112551] tier4_isx021 42-001c: [tier4_isx021_write_reg] : I2C write failed at 0xFFFF=[0x02]
Mar 21 12:23:56 localhost kernel: [  257.124610]  xt_comment(E) xt_state(E) xt_conntrack(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) lzo_rle(E) nf_defrag_ipv4(E) lzo_compress(E) libcrc32c(E) zram(E) iptable_filter(E) micrel(E) ramoops(E) reed_solomon(E) loop(E) tier4_imx490(E) tier4_gw5300(E) nvgpu(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_dspk(E) fuse(E) snd_soc_tegra210_iqc(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_afc(E) snd_soc_tegra210_dmic(E) aes_ce_blk(E) crypto_simd(E) snd_soc_tegra210_admaif(E) cryptd(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) snd_soc_tegra210_mixer(E) snd_soc_tegra_pcm(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_sfc(E) aes_ce_cipher(E) ghash_ce(E) sha2_ce(E) sha256_arm64(E) sha1_ce(E) snd_soc_tegra210_adsp(E) snd_soc_tegra_utils(E) tier4_isx021(E) input_leds(E) binfmt_misc(E) snd_hda_codec_hdmi(E) snd_soc_spdif_tx(E) pwm_fan(E) snd_soc_simple_card_utils(E) snd_hda_tegra(E) nct1008(E) tier4_fpga(E) snd_hda_codec(E)
Mar 21 12:23:56 localhost kernel: [  257.124675]  ina3221(E) userspace_alert(E) nvadsp(E) tier4_max9295(E) tier4_max9296(E) cti_ptn5150(E) snd_soc_tegra210_ahub(E) i2c_nvvrs11(E) snd_hda_core(E) tegra_bpmp_thermal(E) typec(E) tegra210_adma(E) spi_tegra114(E) nvidia(OE) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
Mar 21 12:23:56 localhost kernel: [  257.124699] CPU: 1 PID: 4162 Comm: vi-output, tier Tainted: G           OE     5.10.120-tegra #1
Mar 21 12:23:56 localhost kernel: [  257.124700] Hardware name: Unknown CTI Forge + Orin AGX/CTI Forge + Orin AGX, BIOS 202210.3-52cefd4-dirty 05/27/2024
Mar 21 12:23:56 localhost kernel: [  257.124702] pstate: 60c00009 (nZCv daif +PAN +UAO -TCO BTYPE=--)
Mar 21 12:23:56 localhost kernel: [  257.124705] pc : tegra_capture_ivc_notify_chan_id+0x18c/0x1b0
Mar 21 12:23:56 localhost kernel: [  257.124706] lr : tegra_capture_ivc_notify_chan_id+0x18c/0x1b0
Mar 21 12:23:56 localhost kernel: [  257.124707] sp : ffff80001fdd3ad0
Mar 21 12:23:56 localhost kernel: [  257.124708] x29: ffff80001fdd3ad0 x28: 0000000000000000 
Mar 21 12:23:56 localhost kernel: [  257.124711] x27: 0000000000001043 x26: ffff606481201000 
Mar 21 12:23:56 localhost kernel: [  257.124714] x25: ffff6064b68c5db8 x24: ffff606484d64088 
Mar 21 12:23:56 localhost kernel: [  257.124716] x23: 0000000000000051 x22: ffff606484d64080 
Mar 21 12:23:56 localhost kernel: [  257.124718] x21: ffff606484d64aa0 x20: 0000000000000001 
Mar 21 12:23:56 localhost kernel: [  257.124721] x19: 0000000000000051 x18: 0000000000000000 
Mar 21 12:23:56 localhost kernel: [  257.124723] x17: 0000000000000000 x16: ffffa02050635220 
Mar 21 12:23:56 localhost kernel: [  257.124726] x15: ffff60648a1c0570 x14: ffffffffffffffff 
Mar 21 12:23:56 localhost kernel: [  257.124728] x13: ffffa020526fede8 x12: ffffa020526fea1b 
Mar 21 12:23:56 localhost kernel: [  257.124730] x11: 071c71c71c71c71c x10: 0000000000000ab0 
Mar 21 12:23:56 localhost kernel: [  257.124733] x9 : ffff80001fdd3ad0 x8 : 2073692031207461 
Mar 21 12:23:56 localhost kernel: [  257.124735] x7 : 20747865746e6f63 x6 : 00000000439da0ce 
Mar 21 12:23:56 localhost kernel: [  257.124737] x5 : ffff6073aced0958 x4 : 00000000fffff60b 
Mar 21 12:23:56 localhost kernel: [  257.124740] x3 : 0000000000000001 x2 : ffff6073aced0960 
Mar 21 12:23:56 localhost kernel: [  257.124742] x1 : 0000000000000000 x0 : 0000000000000000 
Mar 21 12:23:56 localhost kernel: [  257.124745] Call trace:
Mar 21 12:23:56 localhost kernel: [  257.124747]  tegra_capture_ivc_notify_chan_id+0x18c/0x1b0
Mar 21 12:23:56 localhost kernel: [  257.124751]  vi_capture_setup+0x35c/0x670
Mar 21 12:23:56 localhost kernel: [  257.124755]  tegra_channel_capture_setup+0xe8/0x2d0
Mar 21 12:23:56 localhost kernel: [  257.124757]  vi5_channel_error_recover+0x154/0x230
Mar 21 12:23:56 localhost kernel: [  257.124758]  tegra_channel_error_recover+0x78/0xa0
Mar 21 12:23:56 localhost kernel: [  257.124760]  tegra_channel_kthread_capture_dequeue+0x130/0x1f0
Mar 21 12:23:56 localhost kernel: [  257.124764]  kthread+0x148/0x170
Mar 21 12:23:56 localhost kernel: [  257.124767]  ret_from_fork+0x10/0x24
Mar 21 12:23:56 localhost kernel: [  257.124768] ---[ end trace ffb032297462fbc6 ]---
Mar 21 12:23:56 localhost kernel: [  257.129373] tegra194-vi5 13e40000.host1x:vi1@14c00000: failed to update control callback
Mar 21 12:23:56 localhost kernel: [  257.129406] (NULL device *): vi_capture_control_message: NULL VI channel received

I tried to apply the 3 patches from Capture-ivc: fix multi-cam race condition without any success.

do you have any clue on how to solver my problem ?

Thanks !
kern_crash.log (24.3 KB)

carolyuu · March 25, 2025, 12:30pm

Hi,

For the camera basic functionality first needs to check the device and driver configuration.
You can reference to below program guide for the detailed information of device tree and driver implementation.

Please refer to Applications Using V4L2 IOCTL Directly by using V4L2 IOCTL to verify basic camera functionality.

Once confirm the configure and still failed below link help to get log and some information and some tips for debug.

Thanks!

ShaneCCC · March 26, 2025, 5:04am

Apply the patch from below link to verify again.

gregoire.zeisser2 · March 26, 2025, 8:26am

I compiled the kernel with these 3 patches but the problem is sitll present

ShaneCCC · March 26, 2025, 8:37am

Please also apply the RCE firmware from below link.

gregoire.zeisser2 · March 26, 2025, 2:36pm

I tried and it still crashes.

I have additional call stack :

tegra_capture_ivc_notify_chan_id
refcount_warn_saturate
vi_capture_release
tegra_capture_ivc_unregister_capture_cb
tegra_capture_ivc_unregister_control_cb

Here is an extract of the kernel log.
kern_crash.log (1020.8 KB)

gregoire.zeisser2 · March 28, 2025, 11:03am

I tried without gstreamer, using v4l2-ctl and I still get the crash


#!/bin/bash

DELAY_SEC=0
STREAM_COUNT=300 # Short or long acquisition.
FORMAT=UYVY # UYVY or NV16

for i in $(seq 0 7);
do
    echo "camera$i $STREAM_COUNT:"
    v4l2-ctl -d /dev/video"$i" -v pixelformat=$FORMAT,width=1920,height=1280 --stream-count "$STREAM_COUNT" --stream-mmap --log-status &
    sleep $DELAY_SEC
done

wait
echo "done"

the problem seems to be on the drivers side

Topic		Replies	Views
All cameras stop work suddenly on orin&jetpack 5.1.3 Jetson AGX Orin camera	9	30	March 6, 2025
When the camera is disconnected or not connected to orin, kernel panic may occur, causing the system to reset Jetson AGX Orin camera	23	99	February 12, 2025
Kernel panic with camera Jetson AGX Orin camera	13	129	November 25, 2024
Using a script to open multiple cameras at the same time causes crash Jetson AGX Orin camera	5	451	June 14, 2023
AGX Orin camera issues Jetson AGX Orin camera , vi	26	946	July 1, 2024
V4L2-CTRL report err_data 64 Jetson AGX Orin camera	6	17	March 5, 2025
Orin 36.3.0 intermittently crashes after cameras failure Jetson AGX Orin camera	4	96	August 2, 2024
Orin R35.1 kernel crash when capture camera image failed Jetson AGX Orin camera	30	2219	January 13, 2023
When I tested the camera, a kernel exception occurred Jetson Orin NX camera	7	205	June 28, 2024
Kernel panic when closing a stuck camera Jetson AGX Orin camera	31	947	June 11, 2024

Jetson orin shutdown caused by multi camera application

Related topics