Hi Folks -
We are having problems with our camera system after booting a module using rcm-boot.
We are using AGX Xavier with RCM boot and we have been having some difficulty with the RTCPU.
The setup is quite simple. We flash a xavier from an x86 host:
./flash.sh neutralino-nsv201 mmcblk0p1
and check that everything on the xavier and our custom carrier board works; everything works.
Next we power cycle the Xavier into RCM mode (either with FORCE_RECOVERY_N or with “reboot --force forced-recovery”). Then we trigger a boot process with
./flash.sh --rcm-boot neutralino-nsv201 mmcblk0p1
The boot process succeeds, but the Xavier console reports a set of errors associated with the rtcpu that look like:
[ 11.247680] tegra186-cam-rtcpu bc00000.rtcpu: Trace buffer configured at IOVA=0xbff00000
[ 13.289141] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: response timeout
[ 13.289306] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 1/2
[ 15.305148] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[ 15.305333] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 2/2
[ 17.321153] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[ 17.321713] tegra-ivc ivc-bc00000.rtcpu: region 0: iova=0xbfec0000-0xbfedffff size=131072
[ 17.322161] tegra-ivc ivc-bc00000.rtcpu:echo@0: echo: ver=0 grp=1 RX[16x64]=0x1000-0x1480 TX[16x64]=0x1480-0x1900
[ 17.322479] tegra-ivc ivc-bc00000.rtcpu:dbg@1: dbg: ver=0 grp=1 RX[1x384]=0x1900-0x1b00 TX[1x384]=0x1b00-0x1d00
[ 17.322751] tegra-ivc ivc-bc00000.rtcpu:dbg@2: dbg: ver=0 grp=1 RX[1x8192]=0x1d00-0x3d80 TX[1x8192]=0x3d80-0x5e00
[ 17.323244] tegra-ivc ivc-bc00000.rtcpu:ivccontrol@3: ivccontrol: ver=0 grp=1 RX[64x320]=0x5e00-0xae80 TX[64x320]=0xae80-0xff00
[ 17.323487] tegra-ivc ivc-bc00000.rtcpu:ivccapture@4: ivccapture: ver=0 grp=1 RX[512x64]=0xff00-0x17f80 TX[512x64]=0x17f80-0x20000
[ 17.323870] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (71)
[ 17.323995] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful
[ 17.324123] tegra186-cam-rtcpu bc00000.rtcpu: fw version 0 has no sha1
[ 19.337186] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x06bfec00: empty mailbox timeout
[ 19.337353] tegra-hsp-mailbox ivc-bc00000.rtcpu: IOVM setup error: 110
[ 19.337473] tegra186-cam-rtcpu bc00000.rtcpu: ivc-bus boot sync failed: -5
[ 19.337598] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=0 sha1=0000000000000000000000000000000000000000
Later, when we attempt to capture images from our cameras, we get similar errors on the console:
[ 1369.632778] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[ 1369.632969] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 1/2
[ 1369.633371] CPU:0, Error:SCE-NOC@0xb600000,irq=493
<... snip ...>
[ 1369.706306] CPU:0, Error:RCE-NOC@0xbe00000,irq=491
<... snip ...>
[ 1369.850729] CPU:0, Error:CBB-NOC@0x2300000,irq=485
<... snip ...>
[ 1372.032671] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[ 1372.032895] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 2/2
[ 1372.033356] CPU:0, Error:SCE-NOC@0xb600000,irq=493
<... snip ...>
[ 1372.106390] CPU:0, Error:RCE-NOC@0xbe00000,irq=491
<... snip ...>
[ 1372.250332] CPU:0, Error:CBB-NOC@0x2300000,irq=485
<... snip ...>
[ 1374.432560] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[ 1374.433270] ------------[ cut here ]------------
[ 1374.433417] WARNING: CPU: 0 PID: 8925 at /build/git/kernel-l4t-r32.6.1/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:96 tegra_capture_ivc_tx+0x11c/0x150
[ 1374.433682] Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter zram overlay bluedroid_pm userspace_alert nvgpu ip_tables x_tables
<... snip ...>
[ 1374.434364] tegra194-vi5 15c10000.vi: IVC control submit failed
[ 1374.434537] tegra194-vi5 15c10000.vi: vi capture setup failed
The complete log from a failed rcm-boot session is here:
rcmboot_camera_fail_dmesg.txt (65.4 KB)
The complete log from a successful emmc boot session on the same hardware is here:
emmcboot_camera_success.txt (61.2 KB)
We did a line-by-line comparison of the console for the exact same hardware booting successfully from eMMC and with this issue using rcm-boot. The only difference in the log files occurs when they get to the rtcpu lines. The successful line reports the rtcpu firmware version as cpu=rce cmd=5 and includes a sha1. The failed version does not.
It seems that the RTCPU is not getting firmware.
Our board configuration is used for our device tree and pinmux; it does not change any of the other parameters. The .conf file in it’s entirety:
source "${LDK_DIR}/p2972-0000.conf.common";
PINMUX_CONFIG="jetson_agx_xavier_nsv201.cfg";
DTB_FILE=neutralino-nsv201.dtb;
TBCDTB_FILE=neutralino-nsv201.dtb;
Most of our testing is with L4T 32.6.1. We also saw the problem with L4T 35.1 but have much less test data.
How does the RTCPU get its firmware? Is this one of the blobs sent by tegraflash that isn’t getting sent in this configuration?
As always, your help and advice is very welcome,
-s