RCE fails to respond to a CAMRTC_HSP_FW_HASH Request through the IVC/HSP

Hello, we have a custom board based on a Xavier NX SoM.

During the boot process the rtcpu-rce driver sets up an IVC/HSP channel used for IPC to the Realtime Camera Engine. During the rtcpu-rce driver probe the driver pushes a message into the channel instructing the RCE to load its firmware, however the HSP never gets a response from the RCE, which results in several hsp-vm1 timeouts. The rtcpu-rce driver attempts to reset the RCE several times but never gets a response from the RCE, which results in a boot sync failure.

Xavier NX TRM doesn’t provide any RCE documentation.

Flash logs are attached.

Here’s a snippet from the boot logs:

[    4.674209] tegra186-cam-rtcpu bc00000.rtcpu: Adding to iommu group 4
[    4.675435] tegra186-cam-rtcpu bc00000.rtcpu: Trace buffer configured at IOVA=0xbff00000
[    6.693859]  bc00000.rtcpu:hsp-vm1: request 0x40b85d43: response timeout
[    8.709858]  bc00000.rtcpu:hsp-vm1: request 0x4077fb32: response timeout
[    8.710069] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 2/3
[   10.725876]  bc00000.rtcpu:hsp-vm1: request 0x4039815a: response timeout
[   10.726076] tegra186-cam-rtcpu bc00000.rtcpu: rce full reset, retry 3/3
[   12.741856]  bc00000.rtcpu:hsp-vm1: request 0x40fad020: response timeout
[   12.747682] tegra-ivc-bus bc00000.rtcpu:ivc-bus: region 0: iova=0xbfec0000-0xbfee01ff size=131584
[   12.748250] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: echo: ver=0 grp=1 RX[16x64]=0x1000-0x1480 TX[16x64]=0x1480-0x1900
[   12.749357] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: dbg: ver=0 grp=1 RX[1x448]=0x1900-0x1b40 TX[1x448]=0x1b40-0x1d80
[   12.750211] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: dbg: ver=0 grp=1 RX[1x8192]=0x1d80-0x3e00 TX[1x8192]=0x3e00-0x5e80
[   12.751867] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccontrol@3: ivccontrol: ver=0 grp=1 RX[64x320]=0x5e80-0xaf00 TX[64x320]=0xaf00-0xff80
[   12.752526] tegra-ivc-bus bc00000.rtcpu:ivc-bus:ivccapture@4: ivccapture: ver=0 grp=1 RX[512x64]=0xff80-0x18000 TX[512x64]=0x18000-0x20080
[   12.752941] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: diag: ver=0 grp=1 RX[1x64]=0x20080-0x20140 TX[1x64]=0x20140-0x20200
[   12.754254] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (48)
[   12.754557] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful

sting_mbu_flash_logs.txt (629.9 KB)

My appologies, the snippet was missing a few critical logs:


[   12.754254] tegra186-cam-rtcpu bc00000.rtcpu: using cam RTCPU IRQ (48)
[   12.754557] tegra186-cam-rtcpu bc00000.rtcpu: tegra_camrtc_mon_create is successful
[   14.757913]  bc00000.rtcpu:hsp-vm1: request 0x46000000: response timeout
[   14.758166]  bc00000.rtcpu:hsp-vm1: FW_HASH failed: 0xffffff92
[   16.773921]  bc00000.rtcpu:hsp-vm1: request 0x44bff000: response timeout
[   16.774191] tegra186-cam-rtcpu bc00000.rtcpu: RTCPU trace: IOVM setup error: -110
[   16.774358] tegra186-cam-rtcpu bc00000.rtcpu: trace boot sync failed: -5
[   18.789919]  bc00000.rtcpu:hsp-vm1: request 0x44bfef80: response timeout
[   20.805930]  bc00000.rtcpu:hsp-vm1: request 0x44bfec00: response timeout
[   20.806196] tegra-ivc-bus bc00000.rtcpu:ivc-bus: IOVM setup error: -110
[   20.806346] tegra186-cam-rtcpu bc00000.rtcpu: ivc-bus boot sync failed: -5
[   20.806634] tegra186-cam-rtcpu bc00000.rtcpu: firmware version cpu=rce cmd=0 sha1=0000000000000000000000000000000000000000

Please provide the version and reproduce step.

Thanks

We are using a Xavier NX SoM with eMMC (p3668-0001). A decompiled version of our DTB is attached as well as the config files and XML flash files. I’ve also attached a file called install.log which contains the output diagnostics from the flash tools.

After flashing the rtcpu-rce fails to load and start it’s firmware. The rtcpu-rce fails to respond to the hsp-vm1 FW_HASH command resulting in the following errors in the logs:

bc00000.rtcpu:hsp-vm1: request 0x44bff000: response timeout
tegra186-cam-rtcpu bc00000.rtcpu: RTCPU trace: IOVM setup error: -110
tegra186-cam-rtcpu bc00000.rtcpu: trace boot sync failed: -5

The bootloader seems to be writing the RCE firmware properly. Is there a way to check the RCE firmware?

sting_mbu.dts (380.8 KB)
jetson-xavier-nx-sting-chassis-carrier.conf (2.5 KB)
sting-chassis-carrier-p3668.conf.common (6.3 KB)
nv_boot_control.conf (118 Bytes)
extlinux.conf (948 Bytes)
flash.xml (40.6 KB)
flash_mb1_fix.xml (40.3 KB)
flash_l4t_t194_spi_emmc_p3668.xml (40.2 KB)

Did you try flash the jetson-xavier-nx-devkit-emmc.conf?

We are using L4T version r35.1. We use the jetson-xavier-nx-sting-chassis-carrier.conf and sting-chassis-carrier-p3668.conf.common which are based on the DevKit’s jetson-xavier-nx-devkit-emmc.conf and p3668.conf.common files.

Our .conf files use a smaller ROOTFS and add a few more partitions. The diffs between the sting-chassis-carrier-p3668.conf.common and jetson-xavier-nx-devkit-emmc.conf are:

bchaco@R9-10EDXJ:sting_dev$ diff jetson-xavier-nx-sting-chassis-carrier.conf jetson-xavier-nx-devkit-emmc.conf
34c34
< source "${LDK_DIR}/sting-chassis-carrier-p3668.conf.common";
---
> source "${LDK_DIR}/p3668.conf.common";
36,37c36
< DTB_FILE=tegra194-p3668-0001-p3509-0000-sting-chassis-carrier.dtb;
< TBCDTB_FILE=tegra194-p3668-0001-p3509-0000-sting-chassis-carrier.dtb;
---
> DTB_FILE=tegra194-p3668-0001-p3509-0000.dtb;

The diffs between the sting-chassis-carrier-p3668.conf.common file and original p3668.conf.common file are:

 diff sting-chassis-carrier-p3668.conf.common p3668.conf.common
101a102,103
> DTB_FILE=tegra194-p3668-all-p3509-0000.dtb;
> TBCDTB_FILE=tegra194-p3668-all-p3509-0000.dtb;
104,109c106
< ROOTFSSIZE=8GiB;
< HOMEFSSIZE=1GiB;
< VARFSSIZE=250MiB;
< VARLFSSIZE=250MiB;
< VARLAFSSIZE=500MiB;
< RECOVERYFSSIZE=4GiB;
---
> ROOTFSSIZE=14GiB;
150c147
< PINMUX_CONFIG="tegra19x-sting-chassis-carrier-pinmux.cfg";
---
> PINMUX_CONFIG="tegra19x-mb1-pinmux-p3668-a01.cfg";
152c149
< PMC_CONFIG="tegra19x-sting-chassis-carrier-padvoltage.cfg";
---
> PMC_CONFIG="tegra19x-mb1-padvoltage-p3668-a01.cfg";

The attached flash_l4t_t194_spi_emmc_p3668.xml has been modified from the Xavier NX devkit version to include additional partitions, with the ‘APP’ partition renamed to ‘ROOT’. The diffs are

--- a/bootloader/t186ref/cfg/flash_l4t_t194_spi_emmc_p3668.xml
+++ b/bootloader/t186ref/cfg/flash_l4t_t194_spi_emmc_p3668.xml
@@ -496,16 +496,14 @@
               partitions defined after this entry are configured in the kernel, and are
               accessible by standard partition tools such as gdisk and parted. </description>
         </partition>
-        <partition name="APP" type="data">
+        <partition name="ROOT" type="data">
             <allocation_policy> sequential </allocation_policy>
             <filesystem_type> basic </filesystem_type>
-            <size> APPSIZE </size>
+            <size> ROOTSIZE </size>
             <file_system_attribute> 0 </file_system_attribute>
             <allocation_attribute> 0x8 </allocation_attribute>
             <align_boundary> 4096 </align_boundary>
             <percent_reserved> 0 </percent_reserved>
-            <filename> APPFILE </filename>
-            <unique_guid> APPUUID </unique_guid>
             <description> **Required.** Contains the rootfs. This partition must be defined
               after `primary_GPT` so that it can be accessed as the fixed known special device
               `/dev/mmcblk0p1`. </description>
@@ -532,6 +530,48 @@
             <filename> DTB_FILE </filename>
             <description> **Required.** Chain A; contains kernel device tree blob. </description>
         </partition>
+        <partition name="HOME" type="data">
+            <allocation_policy> sequential </allocation_policy>
+            <filesystem_type> basic </filesystem_type>
+            <size> HOMESIZE </size>
+            <file_system_attribute> 0 </file_system_attribute>
+            <allocation_attribute> 0x8 </allocation_attribute>
+            <percent_reserved> 0 </percent_reserved>
+        </partition>
+        <partition name="VAR" type="data">
+            <allocation_policy> sequential </allocation_policy>
+            <filesystem_type> basic </filesystem_type>
+            <size> VARSIZE </size>
+            <file_system_attribute> 0 </file_system_attribute>
+            <allocation_attribute> 0x8 </allocation_attribute>
+            <percent_reserved> 0 </percent_reserved>
+        </partition>
+        <partition name="VARL" type="data">
+            <allocation_policy> sequential </allocation_policy>
+            <filesystem_type> basic </filesystem_type>
+            <size> VARLSIZE </size>
+            <file_system_attribute> 0 </file_system_attribute>
+            <allocation_attribute> 0x8 </allocation_attribute>
+            <percent_reserved> 0 </percent_reserved>
+        </partition>
+        <partition name="VARLA" type="data">
+            <allocation_policy> sequential </allocation_policy>
+            <filesystem_type> basic </filesystem_type>
+            <size> VARLASIZE </size>
+            <file_system_attribute> 0 </file_system_attribute>
+            <allocation_attribute> 0x8 </allocation_attribute>
+            <percent_reserved> 0 </percent_reserved>
+        </partition>
+        <partition name="RECOVERY" type="data">
+            <allocation_policy> sequential </allocation_policy>
+            <filesystem_type> basic </filesystem_type>
+            <size> RECOVERYSIZE </size>
+            <file_system_attribute> 0 </file_system_attribute>
+            <allocation_attribute> 0x8 </allocation_attribute>
+            <percent_reserved> 0 </percent_reserved>
+            <unique_guid> APPUUID </unique_guid>
+            <filename> RECOVERYFILE </filename>
+        </partition>
         <partition name="reserved_for_chain_A_user" type="data">
             <allocation_policy> sequential </allocation_policy>
             <filesystem_type> basic </filesystem_type>

Thank you!

Could you try flash with default XNX emmc config without any modification.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.