Jetson TX2 Image Based OTA Update R28.2.1 -> R32.6.1 failing

[clifffwv17]

I followed the process described here:
https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/updating_jetson_and_host.html#wwpID0E0UI0HA

I was using standard downloaded R28.2.1/R32.6.1 L4T, sample rootfs and ota tool packages for the test.

I created the recovery image and OTA update payload package on the host. I followed the instructions for the target for decompressing the ota tools, unpacking the payload package and starting the OTA update process with nv_ota_start.sh. I then rebooted, but the OTA update failed. Here is what I saw in the console:

Checking whether device /dev/mmcblk?p1 exist
Looking for OTA work directory on the device(s): /dev/mmcblk0p1
Checking whether device /dev/sd?1 exist
Looking for OTA work directory on the device(s): /dev/sda1
[ 47.835850] EXT4-fs (sda1): VFS: Can’t find ext4 filesystem
[ 47.841678] EXT4-fs (sda1): VFS: Can’t find ext4 filesystem
[ 47.847588] EXT4-fs (sda1): VFS: Can’t find ext4 filesystem
mount: you must specify the filesystem type
Failed to mount /dev/sda1 to /mnt
Checking whether device /dev/nvme?n1p1 exist
Device /dev/nvme?n1p1 does not exist
ota_align_app_part
[ 48.194144] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
mount /dev/mmcblk0p1 /tmp/app_mnt: Success
total 128
drwxr-xr-x 25 root 0 4096 Feb 20 16:08 .
drwxrwxrwx 4 root 0 0 Feb 20 16:15 …
-rw-r–r-- 1 1000 1000 62 May 17 2018 README.txt
drwxr-xr-x 2 root 0 4096 Apr 17 2018 bin
drwxr-xr-x 5 root 0 4096 Feb 20 15:26 boot
drwxr-xr-x 2 root 0 4096 May 3 2016 dev
drwxr-xr-x 137 root 0 12288 Feb 20 15:33 etc
drwxr-xr-x 4 root 0 4096 Jan 6 2017 home
drwxr-xr-x 22 root 0 4096 Feb 20 14:46 lib
drwx------ 2 root 0 16384 Feb 20 15:26 lost+found
drwxr-xr-x 2 root 0 4096 Aug 8 2016 media
drwxr-xr-x 2 root 0 4096 Apr 20 2016 mnt
drwxr-xr-x 3 root 0 4096 Feb 20 14:46 opt
drwxr-xr-x 2 root 0 4096 Feb 20 16:00 ota
drwxr-xr-x 2 root 0 4096 Feb 20 16:08 ota_log
drwxrwxrwx 5 root 0 4096 Feb 20 16:11 ota_work
drwxr-xr-x 2 root 0 4096 Apr 12 2016 proc
drwx------ 3 root 0 4096 May 6 2016 root
drwxr-xr-x 9 root 0 4096 Dec 12 2016 run
drwxr-xr-x 2 root 0 12288 Apr 17 2018 sbin
drwxr-xr-x 2 root 0 4096 Apr 19 2016 snap
drwxr-xr-x 2 root 0 4096 Apr 20 2016 srv
drwxr-xr-x 2 root 0 4096 Feb 5 2016 sys
drwxrwxrwt 7 root 0 4096 Feb 20 16:14 tmp
drwxr-xr-x 11 root 0 4096 May 3 2016 usr
drwxr-xr-x 15 root 0 4096 Feb 20 14:46 var
init_ota_log /tmp/app_mnt/ota_adjust_app_logs
Create log file at /tmp/app_mnt/ota_adjust_app_logs/ota_20230220-161518.log
ota_log_file=/tmp/app_mnt/ota_adjust_app_logs/ota_20230220-161518.log
init_exception_handler /tmp/app_mnt /tmp/app_mnt/ota_adjust_app_logs/ota_20230220-161518.log 0
APP partition has enough free space
ota_backup_customer_files /tmp/app_mnt/ota_work /tmp/app_mnt
Backing up specified files listed in /tmp/app_mnt/ota_work/ota_backup_files_list.txt
files_line=opt/nvidia
tar cvf /tmp/app_mnt/ota_work/backup_files.tar opt/nvidia
opt/nvidia/
opt/nvidia/l4t-usb-device-mode/
opt/nvidia/l4t-usb-device-mode/nv-l4t-usb-device-mode.sh
opt/nvidia/l4t-usb-device-mode/nv-l4t-usb-device-mode.service
opt/nvidia/l4t-usb-device-mode/LICENSE.filesystem.img
opt/nvidia/l4t-usb-device-mode/filesystem.img
opt/nvidia/l4t-usb-device-mode/nv-l4t-usb-device-mode-stop.sh
gzip /tmp/app_mnt/ota_work/backup_files.tar
e2fsck 1.42.13 (17-May-2015)
/dev/mmcblk0p1 has unsupported feature(s): metadata_csum
e2fsck: Get a newer version of e2fsck!
Failed to run “e2fsck -fy /dev/mmcblk0p1”
Failed to run “ota_align_app_part”
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-4.3#

[linuxdev]

A live update from an older release to a newer release did not exist until sometime in the early R32.x series. There was never any ability to perform a live update to a new release in any of the R28.x series. You’d have to actually flash. One possible aid, if you are afraid of losing content, is that you could first clone (the clone could not be used for update/upgrade, but it would make the previous content visible as a loopback partition on either the host or the Jetson, although this does take an enormous amount of disk space).

[clifffwv17]

The concern is not losing content.

There are two problems that prevent the users using conventional USB flashing:

1/ The Jetson modules are fitted on custom boards is custom mechanics that prevent easy access to the micro USB port
2/ Most of the users are not proficient Linux users - they may have a Windows PC but in most cases not a Linux PC
3/ Most of the users will want an easy and reliable upgrade method that ideally does not involve plugging cables in

One advantage is that the units do have an sdcard fitted. Uboot was modified to only boot from this as part of a proprietary upgrade strategy. If I can understand the OTA upgrade method, especially relating to how the boot partition of the eMMC can be repartitioned, then I may be able to create a custom migration solution.

By the way, the link I gave in my original post does indicate that OTA upgrade is compatible with upgrading from R28.2.1. Why should this link indicate this if it is not supported?

I really think that the lack of “in-the-field” upgrade from R28.2 support from Nvidia is a poor outcome - I hope this has been noted!

[linuxdev]

The short answer is that R28.x did not yet have OTA update. The boot content is also quite different from R32.x boot content, so there are no real substitutes for flash. More details follow.

Actual flash is required to go from R28.2.1 to R32.6.1. Flash can occur only in recovery mode. Jetsons do not have a BIOS, and that early boot content is from partitions (in some models that content is in QSPI memory, and in others visible as partitions of eMMC). Without the BIOS there is no ability to “self flash” since it is the BIOS which brings up and directs control to the flash content. This content does not exist on a Jetson.

A Jetson module can be mounted on a different carrier board and flashed with the content for the wrong carrier board. The Jetson would not boot from that “wrong” carrier board, but could then be transferred to the carrier board which content is for booting, and it should boot from that “correct” carrier board.

The actual recovery mode Jetson is a custom USB device, not a mass storage device, and thus it needs a custom USB driver for the flash in recovery mode. That driver is known as the “driver package”, and is one of the things installed from installing and running JetPack/SDK Manager (it can also be installed manually and run from command line if other setup has been completed, but JetPack/SDKM does that for you). The features used during that flash, and the binary executables, work only on a desktop PC architecture Linux o/s. The underlying command line flash software actually works on a number of distributions, but for JetPack/SDKM capable of working with the TX2 releases it should be Ubuntu 18.04. There is no workaround for needing the Linux host PC and physical access locally over USB. If such a host PC does exist at the user’s end though, it is possible for someone else with remote access to the host PC to perform some of the update (this isn’t particularly useful though).

If these are dev kit models, whereby the SD card is on the module itself (and not the carrier board), then boot content and the equivalent of BIOS content in software would be on the QSPI memory on the module itself. If this is an eMMC model, then that early content is on the eMMC. The early boot content needs to be compatible with the SD card content (and location of drivers differ for an SD card on dev kit models versus eMMC models that have an SD card on the carrier board). If it is a custom carrier board, then the default flash software for a dev kit will only work if the device tree is an exact match. Most third party flash software is the dev kit software with modifications to the device tree (one can often use the default flash software with modifications to a few files when using a third party carrier board).

If this is an eMMC model, then in theory the early boot content can be accessed as a partition. However, that content is signed. In cases where the security fuses have not been burned with a key, then it is a NULL key, but it is still signed. There are options to create the flashed partitions with signature, and in theory, should the system be up and running, then dd could be used with correctly signed partitions to update that boot content. However, there is no room for any mistake, and one must have root access. If security fuses are burned, then you might find that using dd does not work and you might once again be forced to use recovery mode.

Someone else would have to tell you about customizing for OTA upgrade, but until you actually flash an R28.x release to a sufficiently new R32.x release, this cannot occur. In fact the early boot content is so different that I don’t think that using dd with signed partitions would work (imagine if partition layout and sizes differ). Even if you got that boot content updated, then you’d have the task of live replacement of the rootfs with something entirely different as it runs, without the help of a BIOS (since there is no BIOS).

R28.x is quite old. I’m sure OTA update would have been a good idea, and many people would agree with you, but early development simply did not have this at the start. The 64-bit arm64/aarch64 actually used a 32-bit/64-bit mix compatibility in the R24.x series, and R27.x was purely 64-bit, but might have been considered “testing” since it was brand new. R28.x was the first of the “production” 64-bit. R32.x was in fact a major driver of creating OTA update. However, TX2 is itself quite old, so even the newest is old. R32.x has essentially become the maintenance stage, and no new features are added. You can’t get to R34.x+ until you use Xavier, but this too will soon go into maintenance. Xavier can use R35.x, but once we go beyond this, you’ll need to use the Orin platform.