[RCE] BUG: camera-ip/vi5/vi5.c:415 [vi5_check_falcon_failure] "VI FALCON FAILURE: 0x40000000"

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
,
1

hi nvidia:
what is mean for this log
[RCE] BUG: camera-ip/vi5/vi5.c:415 [vi5_check_falcon_failure] “VI FALCON FAILURE: 0x40000000”

Mar 15 23:04:50 pegasus-ubuntu-2 kernel: [ 44.866333]
Aug 5 13:29:23 pegasus-ubuntu-2 kernel: [ 397.722146] [RCE] BUG: camera-ip/vi5/vi5.c:415 [vi5_check_falcon_failure] “VI FALCON FAILURE: 0x40000000”
Aug 5 13:29:23 pegasus-ubuntu-2 kernel: [ 397.778828] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346122] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346124] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346126] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346130] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346132] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346142] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346151] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346156] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346160] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346164] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346166] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346169] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346172] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346209] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346242] video4linux video5: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346250] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346309] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346356] video4linux video2: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346361] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346396] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346464] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346486] video4linux video6: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346491] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346604] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346629] video4linux video0: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346634] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346656] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346687] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346725] video4linux video9: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346732] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.346909] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.347179] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.347452] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.347815] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.348081] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.348261] video4linux video7: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.349650] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.350981] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.354195] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.354216] video4linux video3: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.354219] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.354869] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.601197] video4linux video4: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.606610] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.606670] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.623359] video4linux video1: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.628626] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.636205] tegra194-vi5 13e40000.host1x:vi0@15c00000: vi_capture_release: control failed, errno 1
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.645189] video4linux video8: vi capture release failed
Aug 5 13:29:26 pegasus-ubuntu-2 kernel: [ 400.650631] tegra-camrtc-capture-vi tegra-capture-vi: fatal: error recovery failed

3

May I know the BSP version?

4

R35.4.1 , JP5.1.2

5

Please apply the attached RCE firmware to get detail.

camera-rtcpu-t234-rce.img.r35.4.1.falcon_dbg.txt (519.3 KB)

1 Like
6

camera-teace.txt (25.4 MB)
kern-log.txt (191.1 KB)

this is camera trace and kernel log
when Temp at about 60C, camera rtcpu go bad

[ 242.505111] [RCE] VM0 deactivating.VM0 activating.BUG: camera-ip/vi5/vi5.c:397 [vi5_check_falcon_failure] “VI FALCON FAILURE: 0x100”
[ 242.543562] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!

7

The problem could be cause by thermal event make memory performance drop cause the issue. Could you have some thermal solution to keep the temperature below 60C to verify to confirm the problem.

Thanks

8

HI nvidia:
I only flash R35.1.0 camera-rtcpu-t234-rce.img, it no use。When I try to only flash R36.3 camera-rtcpu-t234-rce.img, camera cannot open.

9

@ShaneCCC
You can find the similar issue / topic in below NV link.
VI FALCON failure
We can confrim thate if ORIN CPU (read from JTOP) below 60C than we can not find any VI FALCON failure.
And we keep everythings same only remove FAN / stop FAN let ORIN CPU temperature rise to 70C.
We can replicate VI FALCON failure issue.
R35.1.0 ===》OK / Temperature Change can not cause VI FALCON FAILURE.
R35.3.1 ===》FAIL / Temperature Change cause VI FALCON FAILURE.
R35.4.1 ===》FAIL / Temperature Change cause VI FALCON FAILURE.

10

hi nvidia:
our camera seting is
12 8M yuv camera

12

May I know what the temperature will go for without FAN while running this case?

13

@ShaneCCC
In my test case, we focus on Orin CPU temperature rise from 50C to 70C in lab room.

Thanks
Yutai

14

We test servel cycle, when temperature > 70 we will use FAN let temperature cooling down to 50C, Then run other cycle again.

15

Thanks for your information.
We are checking if any thermal solution for this case.

Thanks

16

Thanks

17

Could you check if any OC event while the problem showing.

Jetson Orin Nano Series, Jetson Orin NX Series and Jetson AGX Orin Series — Jetson Linux Developer Guide documentation (nvidia.com)

18

@ShaneCCC
We don’t find OC event.
From Orin JTOP info the total power ~30W

19

Please help to apply this FW to check.

The version should be:
tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=c0d3ca5e4fcb15760fb8525f7a25cfb9878b4139

camera-rtcpu-t234-rce.img.0816 (519.3 KB)

Thanks

20

Thanks, we will try and give you feedback later.