Jetson Xavier NX Development Kit: OTA with rootfs redunancy boots with wrong rootfs

I’m using R32.7.2, and a bare Jetson Xavier NX Development Kit, with a 32 GB SDCard.

I want to do an OTA update without version upgrade. I just want to change the content of the rootfs.

Summarized, what I did:

  • Flashed the device with ROOTFS_AB=1

  • Created ota_payload_package

  • Copied the ota_payload_package and the ota_tools to the device

  • Ran nv_ota_start.sh on the device

The script finished with OTA update is completed, device will boot to the slot that is updated.

THE PROBLEM: On the next start, the kernel still has mmcblk0p1 mounted as its root partition, altough nvbootctrl says that slot 1 is active.

It seems that CBOOT passes wrong kernel command line. Especially, rootfs is the wrong one.


[0010.239] I> Linux Cmdline: console=ttyTCU0,115200 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 video=tegrafb earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt rootfs.slot_suffix=_b usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=6 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1

CBOOT should have passed the rootfs UUID, but passes /dev/mmcblk0p1, which is APP and not APP_b.

Further analysis: CBOOT takes kernel command line from the new active kernel partition, which contains exactly the a.m. command line. The kernel partition is written by nv_ota_update_without_layout_change.sh:update_rootfs_with_UE() in this section:


#

# When ROOTFS A/B is enabled, the images update on kernel/kernel-dtb partitions

# are tied to rootfs update. So we need to run "nv_bootloader_payload_updater"

# with individual partition update option to update specific partition.

#

# When ROOTFS A/B is disabled, the images update on kernel/kernel-dtb partitions

# are done along with bootloader update (in function update_bootloader_with_UE).

#

# Update kernel and kernel-dtb partition

local _nv_bootloader_payload_updater=

_nv_bootloader_payload_updater="$(which nv_bootloader_payload_updater)"

cp -f "${kernel_only_image}" "/opt/ota_package/bl_update_payload"

if ! eval "${_nv_bootloader_payload_updater}" --part "kernel${suffix}"; then

ota_log "Failed to run \"${_nv_bootloader_payload_updater} --part kernel${suffix}\""

return 1

fi

if ! eval "${_nv_bootloader_payload_updater}" --part "kernel-dtb${suffix}"; then

ota_log "Failed to run \"${_nv_bootloader_payload_updater} --part kernel-dtb${suffix}\""

return 1

fi

I don’t understand how this can work: The ota_payload_package contains a single kernel_only_payload, which is flashed to the new active kernel partition via nv_bootloader_payload_updater. Don’t you need two different kernel images depending on the slot? Because the kernel image contains the command line, including the rootfs partition UUID?


What I did in detail:


wget https://developer.nvidia.com/embedded/l4t/r32_release_v7.2/t186/jetson_linux_r32.7.2_aarch64.tbz2

wget https://developer.nvidia.com/embedded/l4t/r32_release_v7.2/t186/tegra_linux_sample-root-filesystem_r32.7.2_aarch64.tbz2

tar xpf jetson_linux_r32.7.2_aarch64.tbz2

cd ./Linux_for_Tegra/rootfs

sudo tar -jxpf ../../tegra_linux_sample-root-filesystem_r32.7.2_aarch64.tbz2

cd ../..

wget https://developer.nvidia.com/embedded/l4t/r32_release_v7.2/t186/ota_tools_r32.7.2_aarch64.tbz2

tar xpf ota_tools_r32.7.2_aarch64.tbz2

cd Linux_for_Tegra/

sudo ./apply_binaries.sh

sudo ROOTFS_AB=1 ./flash.sh jetson-xavier-nx-devkit mmcblk0p1

When the device started up, I entered default values in the configuration dialog.

Clone running file system:


sudo ./flash.sh -r -k APP -G my-jetson-fs jetson-xavier-nx-devkit mmcblk0p1

mkdir -p ../tmp-fs

sudo mount -o loop my-jetson-fs.raw ../tmp-fs

Create OTA package:


export TARGET_BSP=/home/klaus/work/R32.7.2/Linux_for_Tegra

export BASE_BSP=/home/klaus/work/R32.7.2/Linux_for_Tegra

sudo ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-xavier-nx-devkit R32-6 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP}

( cd ../tmp-fs && sudo tar -cvpzf ../image_fs.tar.gz --exclude=./data --exclude=./image_fs.tar.gz --one-file-system ./ )

cp tools/ota_tools/version_upgrade/nv_ota_rootfs_updater.sh rootfs_updater.sh

sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -sr -o rootfs_updater.sh -f ../image_fs.tar.gz jetson-xavier-nx-devkit R32-6

On Device:


sudo -i

mkfs.ext4 /dev/mmcblk0p12

mkdir -p /data

echo '/dev/mmcblk0p12 /data ext4 defaults 0 0' >> /etc/fstab

mount -a

mkdir -p /data/ota_work /data/ota

export WORKDIR=/data/ota_work

ln -s /data/ota_work/ /ota_work

ln -s /data/ota /ota

On Host


scp bootloader/jetson-xavier-nx-devkit/ota_payload_package.tar.gz <device-ip>:/data/ota

scp ../ota_tools_r32.7.2_aarch64.tbz2 <device-ip>:/data/ota_work

On Device:


cd /data/ota_work

tar xf ota_tools_r32.7.2_aarch64.tbz2

cd Linux_for_Tegra/tools/ota_tools/version_upgrade/

./nv_ota_start.sh /dev/mmcblk0 /ota/ota_payload_package.tar.gz

Hi klaus.popp,

Please refer to the following instructions for details.
Over-the-Air Update - Update with Rootfs Redundancy Enabled
and Setting Up Your File System - Root File System Redundancy

Why your BASE_BSP and TARGET_BSP are the same?
and <bsp_version> is R32-6, not R32-7?

Could you provide the lsblk result and UART console log on your device for further check?

Why your BASE_BSP and TARGET_BSP are the same?

Because I don’t want to upgrade BSP versions. I just want to update the ROOTFS (and possibly the kernel)

and <bsp_version> is R32-6, not R32-7?

It was the only version that was accepted by build_base_recovery_image.sh.

Here is the lsblk output:

root@nvjet:~# lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0          7:0    0    16M  1 loop 
mtdblock0     31:0    0    32M  0 disk 
mmcblk0      179:0    0    30G  0 disk 
├─mmcblk0p1  179:1    0     7G  0 part 
├─mmcblk0p2  179:2    0     7G  0 part /
├─mmcblk0p3  179:3    0    64M  0 part 
├─mmcblk0p4  179:4    0    64M  0 part 
├─mmcblk0p5  179:5    0   448K  0 part 
├─mmcblk0p6  179:6    0   448K  0 part 
├─mmcblk0p7  179:7    0    63M  0 part 
├─mmcblk0p8  179:8    0   512K  0 part 
├─mmcblk0p9  179:9    0   256K  0 part 
├─mmcblk0p10 179:10   0   256K  0 part 
├─mmcblk0p11 179:11   0   100M  0 part 
└─mmcblk0p12 179:12   0  15.7G  0 part /data
zram0        252:0    0 971.7M  0 disk [SWAP]
zram1        252:1    0 971.7M  0 disk [SWAP]
zram2        252:2    0 971.7M  0 disk [SWAP]
zram3        252:3    0 971.7M  0 disk [SWAP]

root@nvjet:~# parted /dev/mmcblk0
GNU Parted 3.2
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: SD SDCIT (sd/mmc)
Disk /dev/mmcblk0: 32.1GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name               Flags
 1      20.5kB  7516MB  7516MB  ext4         APP                msftdata
 2      7516MB  15.0GB  7516MB  ext4         APP_b              msftdata
 3      15.0GB  15.1GB  67.1MB               kernel             msftdata
 4      15.1GB  15.2GB  67.1MB               kernel_b           msftdata
 5      15.2GB  15.2GB  459kB                kernel-dtb         msftdata
 6      15.2GB  15.2GB  459kB                kernel-dtb_b       msftdata
 7      15.2GB  15.2GB  66.1MB               recovery           msftdata
 8      15.2GB  15.2GB  524kB                recovery-dtb       msftdata
 9      15.2GB  15.2GB  262kB                kernel-bootctrl    msftdata
10      15.2GB  15.2GB  262kB                kernel-bootctrl_b  msftdata
11      15.2GB  15.3GB  105MB                RECROOTFS          msftdata
12      15.3GB  32.1GB  16.8GB  ext4         UDA                msftdata

Here are the booloader logs after OTA:
after-ota.log (67.3 KB)

Hi klaus.popp,

There’s a bug related to CBoot.
The detailed information as following
Cboot in 32.7.2 fails to read extlinux.conf - #15 by WayneWWW

Please help using R32.7.1 or R32.7.3 to check if the issue still exists.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.