Kernel panic with camera

[LucaZhong]

kernel-panic.log (723.7 KB)
Dear Sir：
We used Jetson Orin 64G,with JP5.1.2.Recently，we found kernel panic issue,It happened twice a day.The logs have been uploaded，Please help me found the problem.
L4T version：cat /etc/nv_tegra_release

R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug 1 19:57:35 UTC 2023

thanks

[carolyuu]

Hi,

For the camera basic functionality first needs to check the device and driver configuration.
You can reference to below program guide for the detailed information of device tree and driver implementation.
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=programing#sensor-software-driver-programming

Please refer to Applications Using V4L2 IOCTL Directly by using V4L2 IOCTL to verify basic camera functionality.
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=programing#to-run-a-v4l2-ctl-test

Once confirm the configure and still failed below link help to get log and some information and some tips for debug.
https://elinux.org/Jetson/l4t/Camera_BringUp#Steps_to_enable_more_debug_messages

Thanks!

[ShaneCCC]

I didn’t see the panic message from the log?
What’s command to use camera?

[LucaZhong]

Dear Sir：
he error log is approximately on line 5577

Nov 8 14:22:31 control kernel: [ 1118.183479] cpufreq: cpu0,cur:2324000,set:2201600,set ndiv:172
Nov 8 14:36:00 control kernel: [ 1927.883380] cpufreq: cpu4,cur:2081000,set:2201600,set ndiv:172
Nov 8 14:36:03 control kernel: [ 1930.938144] cpufreq: cpu0,cur:2332000,set:2201600,set ndiv:172
Nov 8 14:36:35 control kernel: [ 1962.309435] cpufreq: cpu4,cur:2325000,set:2201600,set ndiv:172
Nov 8 14:39:32 control kernel: [ 2139.763103] cpufreq: cpu0,cur:2068000,set:2201600,set ndiv:172
Nov 8 14:43:25 control kernel: [ 2373.031147] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Nov 8 14:43:25 control kernel: [ 2373.031155] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Nov 8 14:43:25 control kernel: [ 2373.035135] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Nov 8 14:43:25 control kernel: [ 2373.035137] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Nov 8 14:43:25 control kernel: [ 2373.035345] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Nov 8 14:43:25 control kernel: [ 2373.035345] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Nov 8 14:43:25 control kernel: [ 2373.035357] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Nov 8 14:43:25 control kernel: [ 2373.035602] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Nov 8 14:43:25 control kernel: [ 2373.040323] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Nov 8 14:43:25 control kernel: [ 2373.049512] ttyAMA ttyAMA0: 1 input overrun(s)
Nov 8 14:43:25 control kernel: [ 2373.051048] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Nov 8 14:43:26 control kernel: [ 2373.058575] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Nov 8 14:43:26 control kernel: [ 2373.058901] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Nov 8 14:43:26 control kernel: [ 2373.068054] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Nov 8 14:43:26 control kernel: [ 2373.079191] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Nov 8 14:43:26 control kernel: [ 2374.055130] tegra194-vi5 13e40000.host1x:vi1@14c00000: capture control message timed out
Nov 8 14:43:26 control kernel: [ 2374.055133] tegra194-vi5 13e40000.host1x:vi1@14c00000: capture control message timed out
Nov 8 14:43:26 control kernel: [ 2374.055138] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: release channel IVC failed
Nov 8 14:43:26 control kernel: [ 2374.059124] tegra194-vi5 13e40000.host1x:vi1@14c00000: capture control message timed out
Nov 8 14:43:26 control kernel: [ 2374.059126] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: release channel IVC failed
Nov 8 14:43:26 control kernel: [ 2374.059141] ------------[ cut here ]------------
Nov 8 14:43:26 control kernel: [ 2374.059152] WARNING: CPU: 2 PID: 9877 at /home/jd/liu/jp5.1.2/source/kernel/nvidia/drivers/media/platform/tegra/camera/fusa-capture/capture-vi.c:971 vi_capture_release+0x2a8/0x2d0
Nov 8 14:43:26 control kernel: [ 2374.059152] Modules linked in: xdma(OE) fuse(E) nvidia_modeset(OE) xt_conntrack(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) nfnetlink(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) xt_addrtype(E) iptable_filter(E) br_netfilter(E) isx031(E) nv_imx185(E) i2c_mux_pca954x(E) mttcan(E) can_raw(E) can_dev(E) can(E) lzo_rle(E) lzo_compress(E) zram(E) overlay(E) snd_soc_tegra186_asrc(E) ramoops(E) reed_solomon(E) snd_soc_tegra186_dspk(E) snd_soc_tegra210_ope(E) aes_ce_blk(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_iqc(E) crypto_simd(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_afc(E) cryptd(E) microchip(E) snd_hda_codec_hdmi(E) snd_soc_tegra210_dmic(E) aes_ce_cipher(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_amx(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_sfc(E) snd_hda_tegra(E) ghash_ce(E) snd_soc_tegra_machine_driver(E) snd_hda_codec(E) snd_soc_tegra_utils(E) sha2_ce(E) sha256_arm64(E) sha1_ce(E) snd_soc_spdif_tx(E)
Nov 8 14:43:26 control kernel: [ 2374.059202] lan78xx(E) snd_soc_simple_card_utils(E) nvadsp(E) nct1008(E) snd_soc_tegra210_ahub(E) i2c_nvvrs11(E) snd_hda_core(E) tegra_bpmp_thermal(E) userspace_alert(E) loop(E) nvidia(OE) snd_soc_rt5640(E) snd_soc_rl6231(E) ina3221(E) pwm_fan(E) binfmt_misc(E) nvgpu(E) nvmap(E) nfsd(E) ip_tables(E) x_tables(E) [last unloaded: mtd]
Nov 8 14:43:26 control kernel: [ 2374.059224] CPU: 2 PID: 9877 Comm: vi-output, isx0 Tainted: G OE 5.10.120-tegra #9
Nov 8 14:43:26 control kernel: [ 2374.059225] Hardware name: Unknown Jetson AGX Orin Developer Kit/Jetson AGX Orin Developer Kit, BIOS 4.1-33958178 08/01/2023
Nov 8 14:43:26 control kernel: [ 2374.059228] pstate: 40c00009 (nZcv daif +PAN +UAO -TCO BTYPE=–)
Nov 8 14:43:26 control kernel: [ 2374.059229] pc : vi_capture_release+0x2a8/0x2d0
Nov 8 14:43:26 control kernel: [ 2374.059231] lr : vi_capture_release+0x2a8/0x2d0
Nov 8 14:43:26 control kernel: [ 2374.059231] sp : ffff800030f7bbe0
Nov 8 14:43:26 control kernel: [ 2374.059232] x29: ffff800030f7bbe0 x28: 0000000000000000
Nov 8 14:43:26 control kernel: [ 2374.059233] x27: ffff800030deb998 x26: ffff740682abe880
Nov 8 14:43:26 control kernel: [ 2374.059235] x25: ffff740682abe570 x24: ffff73f96194d880
Nov 8 14:43:26 control kernel: [ 2374.059236] x23: 00000000ffffff92 x22: ffff73f92d7bfd80
Nov 8 14:43:26 control kernel: [ 2374.059237] x21: ffff73f7822a20a0 x20: 0000000000000001
Nov 8 14:43:26 control kernel: [ 2374.059239] x19: ffff73f8035de000 x18: 0000000000000000
Nov 8 14:43:26 control kernel: [ 2374.059240] x17: 0000000000000000 x16: 0000000000000000
Nov 8 14:43:26 control kernel: [ 2374.059241] x15: 0000000000000000 x14: 6320657361656c65
Nov 8 14:43:26 control kernel: [ 2374.059243] x13: 72203a657361656c x12: 65725f6572757470
Nov 8 14:43:26 control kernel: [ 2374.059244] x11: 61635f6976203a30 x10: 3030303063343140
Nov 8 14:43:26 control kernel: [ 2374.059245] x9 : 3169763a78317473 x8 : 656c696166204356
Nov 8 14:43:26 control kernel: [ 2374.059247] x7 : 49206c656e6e6168 x6 : 00000000aad0f862
Nov 8 14:43:26 control kernel: [ 2374.059248] x5 : ffff7406acef1958 x4 : 00000000fffff595
Nov 8 14:43:26 control kernel: [ 2374.059249] x3 : 0000000000000001 x2 : 9d4cc2ee4f493900
Nov 8 14:43:26 control kernel: [ 2374.059251] x1 : 9d4cc2ee4f493900 x0 : 0000000000000000
Nov 8 14:43:26 control kernel: [ 2374.059252] Call trace:
Nov 8 14:43:26 control kernel: [ 2374.059254] vi_capture_release+0x2a8/0x2d0
Nov 8 14:43:26 control kernel: [ 2374.059257] vi5_channel_error_recover+0x4c/0x230
Nov 8 14:43:26 control kernel: [ 2374.059259] tegra_channel_error_recover+0x74/0x90
Nov 8 14:43:26 control kernel: [ 2374.059261] tegra_channel_kthread_capture_dequeue+0x124/0x1e0
Nov 8 14:43:26 control kernel: [ 2374.059264] kthread+0x148/0x170
Nov 8 14:43:26 control kernel: [ 2374.059266] ret_from_fork+0x10/0x18
Nov 8 14:43:26 control kernel: [ 2374.059268] —[ end trace 6e20bc486ea31f6b ]—
Nov 8 14:43:27 control kernel: [ 2374.064964] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: release channel IVC failed
Nov 8 14:43:27 control kernel: [ 2374.071963] ttyAMA ttyAMA0: 15 input overrun(s)
Nov 8 14:43:27 control kernel: [ 2374.072010] tegra194-vi5 13e40000.host1x:vi1@14c00000: capture control message timed out
Nov 8 14:43:27 control kernel: [ 2374.072012] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: release channel IVC failed

[LucaZhong]

Currently the bring up I think it is ok.It can run serverial hours with low probability of crash.

[ShaneCCC]

For the kernel panic you can apply the patch from below link.

Capture-ivc: fix multi-cam race condition

Patch to fix kernel panic for multiple camera capture failed. 0001-capture-ivc-fix-multi-cam-race-condition.patch (3.9 KB) 0001-vi5_fops-fix-mem-leak.patch (3.6 KB) 0001-kernel-camera-remove-csi-enent-sync-for-recover.patch (2.0 KB)

[LucaZhong]

HI
The new log like this,error_data 512、erro sync and timeout.Is some config file is wrong in my dts?
discarding frame 0, flags: 0, err_data 512
[RCE] VM0 deactivating.VM0 activating.ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=241837822304 sof_ts=241841610560
tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms

following is the detail:
149.954756] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 512
[ 150.007264] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[ 150.036597] [RCE] VM0 deactivating.VM0 activating.ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=241837822304 sof_ts=241841610560 gerror_code=2 gerror_data=400065 notify_bits=20000”
[ 152.516578] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[ 152.525731] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[ 152.536486] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[ 152.638953] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 256
[ 152.674268] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[ 152.684582] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=244504465920 sof_ts=244508292224 gerror_code=2 gerror_data=400065 notify_bits=20000”
[ 153.766876] cpufreq: cpu4,cur:2086000,set:2201600,set ndiv:172
[ 155.204557] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[ 155.213711] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[ 155.224493] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[ 155.332566] isx031 33-0013: isx031_stop_streaming: Stop streaming
[ 155.333703] bwmgr API not supported
[ 155.334471] isx031 33-0013: isx031_power_off:
[ 157.211536] isx031 33-0013: isx031_power_on: power on
[ 157.213486] bwmgr API not supported
[ 157.213830] isx031 33-0013: isx031_set_mode: mode index:0
[ 157.217452] isx031 33-0013: isx031_start_streaming: start streaming
[ 157.713199] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 256
[ 157.774263] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[ 157.782332] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=249604421888 sof_ts=249608248192 gerror_code=2 gerror_data=400065 notify_bits=20000”
[ 160.452523] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[ 160.461690] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[ 160.472493] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[ 161.733114] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 256
[ 161.774259] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[ 161.828505] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] “General error queue is out of sync with frame queue. ts=253604387456 sof_ts=253608214304 gerror_code=2 gerror_data=400065 notify_bits=20000”

[LucaZhong]

compatible = “isx031”;
clocks = <&bpmp_clks TEGRA234_CLK_EXTPERIPH1>,
<&bpmp_clks TEGRA234_CLK_EXTPERIPH1>;
clock-names = “extperiph1”, “pllp_grtba”;
mclk = “extperiph1”;
reset-gpios = <&tegra_main_gpio CAM1_PWDN GPIO_ACTIVE_HIGH>;
reg = <0x13>;
devnode = “video13”;
physical_w = “15.0”;
physical_h = “12.5”;

				delayed_gain = "true";

				use_sensor_mode_id = "true";
				
				limit_analog_gain = "true";

				post_crop_frame_drop = "0";

				use_decibel_gain = "true";
				mode0 {
					mclk_khz = "24000";
					num_lanes = "4";
					tegra_sinterface = "serial_d";
					vc_id = "3";
					phy_mode = "DPHY";
					discontinuous_clk = "no";
					dpcm_enable = "false";
					cil_settletime = "0";
					
					dynamic_pixel_bit_depth = "16";
					csi_pixel_bit_depth = "16";
					mode_type = "yuv";
					pixel_phase = "yuyv";

					active_w = "1920";
					active_h = "1536";
					readout_orientation = "0";
					line_length = "2200";
					inherent_gain = "1";
					mclk_multiplier = "2";

					pix_clk_hz = "74250000";  
					serdes_pix_clk_hz = "300000000"; 


					gain_factor = "10";
					min_gain_val = "1"; 
					max_gain_val = "480"; 
					step_gain_val = "3"; 
					default_gain ="0";
					min_hdr_ratio = "1";
					max_hdr_ratio = "1";
					framerate_factor = "1000000";
					min_framerate = "1500000"; 
					max_framerate = "30000000"; 
					step_framerate = "1";
					default_framerate = "30000000";
					exposure_factor = "1000000";
					min_exp_time = "30";
					max_exp_time = "660000"; 
					step_exp_time = "1";
					default_exp_time = "33334";
					
					embedded_metadata_height = "0";
				};

[ShaneCCC]

Apply the RCE firmware from below link.

Unable to stream imx334 with v4l2 driver on Nvidia Jetson Xavier NX where the same dts works for tegra driver(nvargs)

Looks like the same error.

[LucaZhong]

Hello ShaneCCC thank you for your quick response.
My platform is Jetson Orin ,but the topic is Jetson Xavier NX.Is the RCE firmware is the same for both Orin and Xavier ?
Another question,what’s difference between Topic268833_JP-512_rce-fw.7z and camera-rtcpu-t234-rce.img.r35.4.1.asynchronous
thanks

[ShaneCCC]

t234 for Orin serials t194 for Xavier serials.

Thanks

[LucaZhong]

HI ShaneCCC
How should I do,I just want to update camera-rtcpu-t234-rec.img only without other modules.
thanks

[ShaneCCC]

Reference to below two method.

Flash a xavier nx 8g jetpack-5.1.2 but got error: Return value 8

hello alexigu01x, we’re able to update rce-fw success on Xavier-NX + r35.4.1 for instance, $ sudo ./flash.sh -r -k rce-fw jetson-xavier-nx-devkit-emmc mmcblk0p1 ... [ 25.5136 ] tegrarcm_v2 --ismb2 [ 25.5231 ] tegradevflash_v2 --reboot coldboot [ 25.5246 ] Bootloader version 01.00.0000 *** The [rce-fw] has been updated successfully. *** it might be an issue with VM environment, you may review USB port forward settings. or… please try with native Ubuntu OS.

Update rce firmware without flash

so, you should running with Updating Jetson Linux with Image-Based Over-the-Air Update for moving whole L4T version to r35.6.0