Jetson Xavier NX Development Kit: OTA with rootfs redunancy boots with wrong rootfs

I’m using R32.7.2, and a bare Jetson Xavier NX Development Kit, with a 32 GB SDCard.

I want to do an OTA update without version upgrade. I just want to change the content of the rootfs.

Summarized, what I did:

  • Flashed the device with ROOTFS_AB=1

  • Created ota_payload_package

  • Copied the ota_payload_package and the ota_tools to the device

  • Ran nv_ota_start.sh on the device

The script finished with OTA update is completed, device will boot to the slot that is updated.

THE PROBLEM: On the next start, the kernel still has mmcblk0p1 mounted as its root partition, altough nvbootctrl says that slot 1 is active.

It seems that CBOOT passes wrong kernel command line. Especially, rootfs is the wrong one.


[0010.239] I> Linux Cmdline: console=ttyTCU0,115200 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 video=tegrafb earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt rootfs.slot_suffix=_b usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=6 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1

CBOOT should have passed the rootfs UUID, but passes /dev/mmcblk0p1, which is APP and not APP_b.

Further analysis: CBOOT takes kernel command line from the new active kernel partition, which contains exactly the a.m. command line. The kernel partition is written by nv_ota_update_without_layout_change.sh:update_rootfs_with_UE() in this section:


#

# When ROOTFS A/B is enabled, the images update on kernel/kernel-dtb partitions

# are tied to rootfs update. So we need to run "nv_bootloader_payload_updater"

# with individual partition update option to update specific partition.

#

# When ROOTFS A/B is disabled, the images update on kernel/kernel-dtb partitions

# are done along with bootloader update (in function update_bootloader_with_UE).

#

# Update kernel and kernel-dtb partition

local _nv_bootloader_payload_updater=

_nv_bootloader_payload_updater="$(which nv_bootloader_payload_updater)"

cp -f "${kernel_only_image}" "/opt/ota_package/bl_update_payload"

if ! eval "${_nv_bootloader_payload_updater}" --part "kernel${suffix}"; then

ota_log "Failed to run \"${_nv_bootloader_payload_updater} --part kernel${suffix}\""

return 1

fi

if ! eval "${_nv_bootloader_payload_updater}" --part "kernel-dtb${suffix}"; then

ota_log "Failed to run \"${_nv_bootloader_payload_updater} --part kernel-dtb${suffix}\""

return 1

fi

I don’t understand how this can work: The ota_payload_package contains a single kernel_only_payload, which is flashed to the new active kernel partition via nv_bootloader_payload_updater. Don’t you need two different kernel images depending on the slot? Because the kernel image contains the command line, including the rootfs partition UUID?


What I did in detail:


wget https://developer.nvidia.com/embedded/l4t/r32_release_v7.2/t186/jetson_linux_r32.7.2_aarch64.tbz2

wget https://developer.nvidia.com/embedded/l4t/r32_release_v7.2/t186/tegra_linux_sample-root-filesystem_r32.7.2_aarch64.tbz2

tar xpf jetson_linux_r32.7.2_aarch64.tbz2

cd ./Linux_for_Tegra/rootfs

sudo tar -jxpf ../../tegra_linux_sample-root-filesystem_r32.7.2_aarch64.tbz2

cd ../..

wget https://developer.nvidia.com/embedded/l4t/r32_release_v7.2/t186/ota_tools_r32.7.2_aarch64.tbz2

tar xpf ota_tools_r32.7.2_aarch64.tbz2

cd Linux_for_Tegra/

sudo ./apply_binaries.sh

sudo ROOTFS_AB=1 ./flash.sh jetson-xavier-nx-devkit mmcblk0p1

When the device started up, I entered default values in the configuration dialog.

Clone running file system:


sudo ./flash.sh -r -k APP -G my-jetson-fs jetson-xavier-nx-devkit mmcblk0p1

mkdir -p ../tmp-fs

sudo mount -o loop my-jetson-fs.raw ../tmp-fs

Create OTA package:


export TARGET_BSP=/home/klaus/work/R32.7.2/Linux_for_Tegra

export BASE_BSP=/home/klaus/work/R32.7.2/Linux_for_Tegra

sudo ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-xavier-nx-devkit R32-6 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP}

( cd ../tmp-fs && sudo tar -cvpzf ../image_fs.tar.gz --exclude=./data --exclude=./image_fs.tar.gz --one-file-system ./ )

cp tools/ota_tools/version_upgrade/nv_ota_rootfs_updater.sh rootfs_updater.sh

sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -sr -o rootfs_updater.sh -f ../image_fs.tar.gz jetson-xavier-nx-devkit R32-6

On Device:


sudo -i

mkfs.ext4 /dev/mmcblk0p12

mkdir -p /data

echo '/dev/mmcblk0p12 /data ext4 defaults 0 0' >> /etc/fstab

mount -a

mkdir -p /data/ota_work /data/ota

export WORKDIR=/data/ota_work

ln -s /data/ota_work/ /ota_work

ln -s /data/ota /ota

On Host


scp bootloader/jetson-xavier-nx-devkit/ota_payload_package.tar.gz <device-ip>:/data/ota

scp ../ota_tools_r32.7.2_aarch64.tbz2 <device-ip>:/data/ota_work

On Device:


cd /data/ota_work

tar xf ota_tools_r32.7.2_aarch64.tbz2

cd Linux_for_Tegra/tools/ota_tools/version_upgrade/

./nv_ota_start.sh /dev/mmcblk0 /ota/ota_payload_package.tar.gz

Hi klaus.popp,

Please refer to the following instructions for details.
Over-the-Air Update - Update with Rootfs Redundancy Enabled
and Setting Up Your File System - Root File System Redundancy

Why your BASE_BSP and TARGET_BSP are the same?
and <bsp_version> is R32-6, not R32-7?

Could you provide the lsblk result and UART console log on your device for further check?

Why your BASE_BSP and TARGET_BSP are the same?

Because I don’t want to upgrade BSP versions. I just want to update the ROOTFS (and possibly the kernel)

and <bsp_version> is R32-6, not R32-7?

It was the only version that was accepted by build_base_recovery_image.sh.

Here is the lsblk output:

root@nvjet:~# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0          7:0    0    16M  1 loop 
mtdblock0     31:0    0    32M  0 disk 
mmcblk0      179:0    0    30G  0 disk 
├─mmcblk0p1  179:1    0     7G  0 part 
├─mmcblk0p2  179:2    0     7G  0 part /
├─mmcblk0p3  179:3    0    64M  0 part 
├─mmcblk0p4  179:4    0    64M  0 part 
├─mmcblk0p5  179:5    0   448K  0 part 
├─mmcblk0p6  179:6    0   448K  0 part 
├─mmcblk0p7  179:7    0    63M  0 part 
├─mmcblk0p8  179:8    0   512K  0 part 
├─mmcblk0p9  179:9    0   256K  0 part 
├─mmcblk0p10 179:10   0   256K  0 part 
├─mmcblk0p11 179:11   0   100M  0 part 
└─mmcblk0p12 179:12   0  15.7G  0 part /data
zram0        252:0    0 971.7M  0 disk [SWAP]
zram1        252:1    0 971.7M  0 disk [SWAP]
zram2        252:2    0 971.7M  0 disk [SWAP]
zram3        252:3    0 971.7M  0 disk [SWAP]

root@nvjet:~# parted /dev/mmcblk0
GNU Parted 3.2
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: SD SDCIT (sd/mmc)
Disk /dev/mmcblk0: 32.1GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name               Flags
 1      20.5kB  7516MB  7516MB  ext4         APP                msftdata
 2      7516MB  15.0GB  7516MB  ext4         APP_b              msftdata
 3      15.0GB  15.1GB  67.1MB               kernel             msftdata
 4      15.1GB  15.2GB  67.1MB               kernel_b           msftdata
 5      15.2GB  15.2GB  459kB                kernel-dtb         msftdata
 6      15.2GB  15.2GB  459kB                kernel-dtb_b       msftdata
 7      15.2GB  15.2GB  66.1MB               recovery           msftdata
 8      15.2GB  15.2GB  524kB                recovery-dtb       msftdata
 9      15.2GB  15.2GB  262kB                kernel-bootctrl    msftdata
10      15.2GB  15.2GB  262kB                kernel-bootctrl_b  msftdata
11      15.2GB  15.3GB  105MB                RECROOTFS          msftdata
12      15.3GB  32.1GB  16.8GB  ext4         UDA                msftdata

Here are the booloader logs after OTA:
after-ota.log (67.3 KB)

Hi klaus.popp,

There’s a bug related to CBoot.
The detailed information as following
Cboot in 32.7.2 fails to read extlinux.conf - #15 by WayneWWW

Please help using R32.7.1 or R32.7.3 to check if the issue still exists.