Problem with Camera and RTCPU after RCM-BOOT

Hi Folks -

We are having problems with our camera system after booting a module using rcm-boot.

We are using AGX Xavier with RCM boot and we have been having some difficulty with the RTCPU.

The setup is quite simple. We flash a xavier from an x86 host:
./flash.sh neutralino-nsv201 mmcblk0p1
and check that everything on the xavier and our custom carrier board works; everything works.

Next we power cycle the Xavier into RCM mode (either with FORCE_RECOVERY_N or with “reboot --force forced-recovery”). Then we trigger a boot process with
./flash.sh --rcm-boot neutralino-nsv201 mmcblk0p1

The boot process succeeds, but the Xavier console reports a set of errors associated with the rtcpu that look like:

[   11.247680] tegra186-cam-rtcpu bc00000.rtcpu: Trace buffer configured at IOVA=0xbff00000
[   13.289141] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: response timeout
[   13.289306] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 1/2
[   15.305148] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[   15.305333] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 2/2
[   17.321153] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[   17.321713] tegra-ivc ivc-bc00000.rtcpu: region 0: iova=0xbfec0000-0xbfedffff size=131072
[   17.322161] tegra-ivc ivc-bc00000.rtcpu:echo@0: echo: ver=0 grp=1 RX[16x64]=0x1000-0x1480 TX[16x64]=0x1480-0x1900
[   17.322479] tegra-ivc ivc-bc00000.rtcpu:dbg@1: dbg: ver=0 grp=1 RX[1x384]=0x1900-0x1b00 TX[1x384]=0x1b00-0x1d00
[   17.322751] tegra-ivc ivc-bc00000.rtcpu:dbg@2: dbg: ver=0 grp=1 RX[1x8192]=0x1d00-0x3d80 TX[1x8192]=0x3d80-0x5e00
[   17.323244] tegra-ivc ivc-bc00000.rtcpu:ivccontrol@3: ivccontrol: ver=0 grp=1 RX[64x320]=0x5e00-0xae80 TX[64x320]=0xae80-0xff00
[   17.323487] tegra-ivc ivc-bc00000.rtcpu:ivccapture@4: ivccapture: ver=0 grp=1 RX[512x64]=0xff00-0x17f80 TX[512x64]=0x17f80-0x20000
[   17.323870] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (71)
[   17.323995] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful
[   17.324123] tegra186-cam-rtcpu bc00000.rtcpu: fw version 0 has no sha1
[   19.337186] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x06bfec00: empty mailbox timeout
[   19.337353] tegra-hsp-mailbox ivc-bc00000.rtcpu: IOVM setup error: 110
[   19.337473] tegra186-cam-rtcpu bc00000.rtcpu: ivc-bus boot sync failed: -5
[   19.337598] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=0 sha1=0000000000000000000000000000000000000000

Later, when we attempt to capture images from our cameras, we get similar errors on the console:

[ 1369.632778] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[ 1369.632969] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 1/2
[ 1369.633371] CPU:0, Error:SCE-NOC@0xb600000,irq=493
<... snip ...>
[ 1369.706306] CPU:0, Error:RCE-NOC@0xbe00000,irq=491
<... snip ...>
[ 1369.850729] CPU:0, Error:CBB-NOC@0x2300000,irq=485
<... snip ...>
[ 1372.032671] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[ 1372.032895] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 2/2
[ 1372.033356] CPU:0, Error:SCE-NOC@0xb600000,irq=493
<... snip ...>
[ 1372.106390] CPU:0, Error:RCE-NOC@0xbe00000,irq=491
<... snip ...>
[ 1372.250332] CPU:0, Error:CBB-NOC@0x2300000,irq=485
<... snip ...>
[ 1374.432560] tegra186-cam-rtcpu bc00000.rtcpu: command: 0x00000000: empty mailbox timeout
[ 1374.433270] ------------[ cut here ]------------
[ 1374.433417] WARNING: CPU: 0 PID: 8925 at /build/git/kernel-l4t-r32.6.1/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:96 tegra_capture_ivc_tx+0x11c/0x150
[ 1374.433682] Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter zram overlay bluedroid_pm userspace_alert nvgpu ip_tables x_tables
<... snip ...>
[ 1374.434364] tegra194-vi5 15c10000.vi: IVC control submit failed
[ 1374.434537] tegra194-vi5 15c10000.vi: vi capture setup failed

The complete log from a failed rcm-boot session is here:
rcmboot_camera_fail_dmesg.txt (65.4 KB)

The complete log from a successful emmc boot session on the same hardware is here:
emmcboot_camera_success.txt (61.2 KB)

We did a line-by-line comparison of the console for the exact same hardware booting successfully from eMMC and with this issue using rcm-boot. The only difference in the log files occurs when they get to the rtcpu lines. The successful line reports the rtcpu firmware version as cpu=rce cmd=5 and includes a sha1. The failed version does not.

It seems that the RTCPU is not getting firmware.

Our board configuration is used for our device tree and pinmux; it does not change any of the other parameters. The .conf file in it’s entirety:

source "${LDK_DIR}/p2972-0000.conf.common";
PINMUX_CONFIG="jetson_agx_xavier_nsv201.cfg";
DTB_FILE=neutralino-nsv201.dtb;
TBCDTB_FILE=neutralino-nsv201.dtb;

Most of our testing is with L4T 32.6.1. We also saw the problem with L4T 35.1 but have much less test data.

How does the RTCPU get its firmware? Is this one of the blobs sent by tegraflash that isn’t getting sent in this configuration?

As always, your help and advice is very welcome,
-s

Hi,
Do you observe the issue on Xavier developer kit? For Xavier developer kit, we have E3326 camera board. Not sure if you have the board. If you have it, please try and see if you can reproduce it on developer kit.

Hi,
We have checked and confirmed this command is valid:

./flash.sh neutralino-nsv201 mmcblk0p1

So that rootfs is flashed to eMMC.

For this command:

./flash.sh --rcm-boot neutralino-nsv201 mmcblk0p1

The bootloader is flashed to DRAM and after rebooting the device, bootloader is gone and need to re-flash the device.

So for flashing system image, please not to set –rcm-boot.

Beside using flash.sh, you may try initrd, please take a look at

nvidia/nvidia_sdk/JetPack_5.0.2_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/tools/kernel_flash/README_initrd_flash.txt

HI @DaneLLL -

We have been able to confirm the problem with the devkit, even without a camera board. There is a very subtle message that shows the difference between success and failure.

Devkit SUCCESSFUL Flash Boot

[Fri Nov 25 22:00:14 2022] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (54)
[Fri Nov 25 22:00:14 2022] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful
[Fri Nov 25 22:00:14 2022] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=6 sha1=d48f1e279e6679b09020947e8de088eabfeaa475

Devkit FAILED RCM Boot

[Fri Nov 25 21:00:22 2022] tegra186-cam-rtcpu bc00000.rtcpu: ivc-bus boot sync failed: -5
[Fri Nov 25 21:00:22 2022] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=0 sha1=0000000000000000000000000000000000000000

This message, “rtcpu firmware version 0” is consistent across all the “failed” RCM boots and for jetpack 4.6 and 5.0. We see it both with the devkit and our custom board.

Our current hypothesis is that the RCM boot sequence does not uplaod RTCPU firmware. Can you confirm this? Is there any other way to upload that firmware?

Thank you!

Hi,
We would suggest use initrd. Please share what use-case you would like to do, so that we can check and suggest the initrd command.

Hi @DaneLLL,

Do you mean to use the initrd after booting off the emmc?

Our use case is to RCM boot a Jetson with a corrupted emmc, so this doesn’t work.

If you mean that we should load the rtcpu firmware from the initrd: that would be great. Is that possible?

Thanks!

Hi,
The use-case looks strange. For Xavier modules, the bootloader has to be flashed to emmc. If the emmc is broken, it cannot be used for production. Is there a reason you have to use Xavier modules with broken emmc?

Indeed, our use case may be strange.

Yes, there are reasons why we have to use modules with broken emmc. Unfortunately, this is not optional.

Do you know if it is possible to load the existing firmware image to the camera-rtcpu after the rcm boot process has completed?

Hi,
The L4T release is for valid modules. For Xavier, we support flashing bootloader and rootfs to eMMC, or bootloader to eMMC and rootfs to external storage(NVMe SSD). If eMMC is broken, the flashing tools(flash.sh and initrd) may not work properly.