Camera rtcpu gone bad!

I have four cameras connected with orin(r35.4 camera_rce_fw).when I call a phone and let it close to the connector, the gst-launch can not show the camera image.DMESG print camera rtcpu gone bad!
/////////////////////////////////////////some logs//////////////////////////////////////////////////
[ 2449.251696] recover start,vir chan id:3…
[ 2449.251697] vi_capture_release,vir chan id:3
[ 2449.252601] (NULL device *): vi_capture_control_message: NULL VI channel received
[ 2449.263591] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel,chanid:6,vichanid:0,virtual_channel_id:2
[ 2449.263592] recover start,vir chan id:2…
[ 2449.263593] vi_capture_release,vir chan id:2
[ 2449.273996] [RCE] *** DATA ABORT EXCEPTION ***
[ 2449.275215] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=2, csi_port=2
[ 2449.288392] [RCE] R0: 799e3185 R1: 00000000 R2: 00000100 R3: 78000000
[ 2449.296057] (NULL device *): vi_capture_control_message: NULL VI channel received
[ 2449.296060] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 2 vc- 3
[ 2449.309320] [RCE] R4: 00000001 R5: 0bcda248 R6: 799e3100 R7: 799e3174
[ 2449.319951] recover,tegra_channel_capture_setup start,virid:3
[ 2449.327567] [RCE] R8: 799e30f4 R9: 00000023 R10: 799e3000 R11: 00000069
[ 2449.327570] [RCE] R12: bf56b000 SP: 0bccfea0 LR: 0bca8978 PC: 0bca8a0c
[ 2449.327574] [RCE] SPSR: 8000001f
[ 2449.338108] [RCE] DFSR: 0000000d DFAR: 000000f8 ADFSR: 00000000
[ 2449.338109] [RCE] IFSR: 00000000 IFAR: 00000000 AIFSR: 00000000
[ 2449.338124] tegra_channel_capture_setup,vi_capture_setup call,virid:3
[ 2449.338138] tegra_capture_ivc_register_control_cb,ctx_id:86
[ 2449.347293] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[ 2449.350003] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[ 2449.359243] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel,chanid:4,vichanid:1,virtual_channel_id:0
[ 2449.372492] recover start,vir chan id:0…
[ 2449.372493] vi_capture_release,vir chan id:0
[ 2449.372508] ------------[ cut here ]------------
[ 2449.372519] WARNING: CPU: 5 PID: 4438 at /home/csu/code/dvpn-poseidon-orin-os/public/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:44 tegra_capture_ivc_tx+0x120/0x150
[ 2449.388791] Modules linked in: nvidia_modeset(OE) fuse(E) xt_state(E) mttcan(E) can_dev(E) xt_conntrack(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) nfnetlink(E) xt_addrtype(E) iptable_filter(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) libcrc32c(E) br_netfilter(E) lzo_rle(E) lzo_compress(E) zram(E) overlay(E) option(E) usb_wwan(E) usbserial(E) GobiNet(E) ramoops(E) reed_solomon(E) bluetooth(E) loop(E) nvgpu(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) ghash_ce(E) sha2_ce(E) input_leds(E) sha256_arm64(E) sha1_ce(E) pwm_fan(E) nct1008(E) ina3221(E) tegra_bpmp_thermal(E) userspace_alert(E) spi_tegra114(E) nvidia(OE) binfmt_misc(E) nvmap(E) ip_tables(E) x_tables(E)
[ 2449.388846] CPU: 5 PID: 4438 Comm: vi-output, ov27 Tainted: G OE 5.10.104-1-1-1-tegra #13
[ 2449.388847] Hardware name: Jetson AGX Orin (DT)
[ 2449.388849] pstate: 60c00009 (nZCv daif +PAN +UAO -TCO BTYPE=–)
[ 2449.388851] pc : tegra_capture_ivc_tx+0x120/0x150
[ 2449.388853] lr : tegra_capture_ivc_tx+0x30/0x150
[ 2449.388854] sp : ffff800020fa3ae0
[ 2449.388855] x29: ffff800020fa3ae0 x28: 0000000000000000
[ 2449.388858] x27: ffff80001ce1b988 x26: 0000000000000000
[ 2449.388860] x25: 0000000000000118 x24: ffff69742a193000
[ 2449.388862] x23: ffff800020fa3c20 x22: ffff6973b85b0e80
[ 2449.388864] x21: ffff697a88371400 x20: 0000000000000118
[ 2449.388866] x19: ffff697a8835a080 x18: 0000000000000001
[ 2449.388868] x17: 0000000000000000 x16: ffffb334637e3210
[ 2449.388870] x15: ffff6973b85b13f0 x14: ffffffffffffffff
[ 2449.388872] x13: ffffb3346563dde8 x12: ffffb3346563da27
[ 2449.388874] x11: 2069762d65727574 x10: 65725f727265203a
[ 2449.388876] x9 : 0000000000000000 x8 : ffff800020fa3d38
[ 2449.388878] x7 : 0000000000000000 x6 : 000000000000003f
[ 2449.388880] x5 : 0000000000000001 x4 : 0000000000000015
[ 2449.388882] x3 : 0000000000000000 x2 : ffffb3346397e090
[ 2449.388883] x1 : 0000000000000000 x0 : 0000000000000000
camera_rce_error.log (134.9 KB)

Can you give more information for below.

Thanks

Make a call and hold the phone close to the connector of cameras. The image will be disturbed due to electromagnetic interference.But sometimes I put away the phone,gst-launch-1.0 will not receive image data.I make sure the cameras have datas.I found two reasons:

  1. Camera rtcpu gone bad!
    2.vi5_fops.c vi_capture_status function timeout
    how to solve the two questions?

Hi, the phone call might cause such RF interference to high speed signals on board since the frequency might be same. You can do some shielding for this if necessary.

I’m sorry. I didn’t make myself clear. You misunderstood me. the phone call might cause such RF interference,that is right,but it is not my question.How can the rtcpu crash when the phone cause such RF interference.The orin camera system can not work well more when I put away my phone,that is the question

Do you work on r35.4.1? It’s Bayer raw sensor? What’s the gstreamer command?

Thanks

I have tested r35.4 and r35.1, they all have the same problem. gmsl camera, max92595+max96712.

gst-launch-1.0 v4l2src device=“/dev/video4” ! “video/x-raw,width=1920,height=1080,format=UYVY” ! xvimagesink -e

Please modify the vi5_fops.c to avoid recover process while CAPTURE_ERROR to try.

 		if (chan->capture_state == CAPTURE_ERROR) {
744  			spin_unlock_irqrestore(&chan->capture_state_lock,
745  				flags);
746  			err = tegra_channel_error_recover(chan, false);
747  			if (err) {
748  				dev_err(chan->vi->dev,
749  					"fatal: error recovery failed\n");
750  				break;
751  			}

the code have added it, it did have some effect.But sometimes it can not recover.
1.if rtcpu gone bad, I think the camera fw was crash.it can not recover
2.when rtcpu has not gone bad,vi5_fops.c vi_capture_status function timeout.Sometimes tegra_channel_error_recover can recover,but sometimes it can not.It’s not 100 percent useful

I mean remove the recover mechanism to check if able to avoid the RTC crash.

Thanks

yes, it can avoid the rtc crash.Rtc crash is one question.Other question,when vi5_fops.c/vi_capture_status function timeout,sometimes gst-launch-1.0 does not return to normal operation

Yes, that could be some corner case would cause that and need to restart terminal and restart the APP.

It’s not the problem of the terminal or gst-launch-1.0, it is confirmed.There are some bugs in nvidia/drivers/media or the rtcpu.

Thanks for your information.
Due to the RCE firmware is passive maybe need to figure it out from vi5.

Thanks

I have solved some bugs from vi5.Now I have one question.How to recover csi? The mipi csi data is ok,but vi5_fops.c vi_capture_status function timeout,when the max96712 stop csi and start csi, it can recover.How to use vi5 to recover that stream? For example write orin register to stop receiving mipi data for a while,and then restart it.

Please reference to vi5_channel_error_recover() in vi5_fops.c for recover.

It can recover some errors,but vi5_fops.c/vi_capture_status function timeout that error sometimes cannot recover.

Suggest restart all pipeline for unable recover case.

Thanks

Can you give me example codes? Thank you.