Remote TX1/TX2 OTA upgrade from Jetpack 3.0 to Jetpack 4.4.1

In case anyone is interested in this, I’ve come up with a dangerous way to do a remote L4T upgrade in the field. “Dangerous” in this case means that if the device loses power or reboots midway through the update, it will likely need to be physically flashed (since it might not boot, or if it does, kernel modules/firmware required may not be present yet as the upgrade didn’t complete).

It’s unsupported by Nvidia (with good reason), but we got it working for our use case.

Proceed at your own risk!

In our case the update was from Jetpack 3.0 (R24.2.1, kernel 3.10.96-tegra, Ubuntu 16.04) to Jetpack 4.4.1 (R32.4.4, kernel 4.9.120 custom, Ubuntu 18.04), but the same principles should hold for upgrades between arbitrary versions and also for upgrading TX2s (though with more partitions).

The first thing needed is to prepare an upgrade zip payload. This should contain:

  • A patched 3.10.96 kernel and associated modules to allow the kernel to expose the bootloader /dev/mmcblk0boot0 and /dev/mmcblk0boot1 block devices
  • The R32.4.4 partition table and images of each partition (taken from a clean JetPack 4.4 flash of a TX1)
  • The R32.4.4 /boot folder containing the new kernel, extlinux and DTBs (taken from a clean JetPack 4.4 flash of a TX1)
  • The jetson-ota-public.key and nvidia-l4t-apt-source.list
  • The R32.4.4 nvidia-l4t-kernel, nvidia-l4t-kernel-dtbs and nvidia-l4t-kernel-headers deb packages (found in Linux_for_Tegra/kernel)
  • A bl_update_payload to verify that the bootloader partition’s checksums match what we expect after a clean flash
  • An apt-clone.tar.gz from a clean (taken from a clean JetPack 4.4 flash of a TX1)

Steps for preparing the upgrade.zip payload:

  1. Using a Jetpack 3.0 on a Ubuntu 14.04 x86 machine, clone the kernel sources using ./source_sync.sh and apply the kernel patch below.

In the case of R24.2.1, the kernel (3.10.96-tegra) doesn’t expose the mmcblk0boot0 and mmcblk0boot1 devices unless patched with the below:

diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
index 0c2bc73..0ef7c9a 100644
--- a/arch/arm64/kernel/vdso32/Makefile
+++ b/arch/arm64/kernel/vdso32/Makefile
@@ -11,7 +11,7 @@ obj-vdso32 := $(addprefix $(obj)/, $(obj-vdso32))
 
 GCOV_PROFILE := n
 
-ccflags-y := -shared -fPIC -fno-common -fno-builtin -march=armv7-a
+ccflags-y := -shared -fPIC -fomit-frame-pointer -fno-common -fno-builtin -march=armv7-a
 ccflags-y += -nostdlib -Wl,-soname=linux-vdso32.so.1 \
 		$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
 asflags-y := -D__VDSO32__ -s
diff --git a/drivers/mmc/host/sdhci-tegra.c b/drivers/mmc/host/sdhci-tegra.c
index 4a9e322..5eb0cd7 100644
--- a/drivers/mmc/host/sdhci-tegra.c
+++ b/drivers/mmc/host/sdhci-tegra.c
@@ -6064,7 +6064,7 @@ static int sdhci_tegra_probe(struct platform_device *pdev)
 	host->mmc->caps |= MMC_CAP_WAIT_WHILE_BUSY;
 
 	/* disable access to boot partitions */
-	host->mmc->caps2 |= MMC_CAP2_BOOTPART_NOACC;
+	//host->mmc->caps2 |= MMC_CAP2_BOOTPART_NOACC;
 
 	if (soc_data->nvquirks & NVQUIRK_ENABLE_HS200)
 		host->mmc->caps2 |= MMC_CAP2_HS200;
diff --git a/drivers/platform/tegra/tegra21_clocks.c b/drivers/platform/tegra/tegra21_clocks.c
index f539d42..f31cdc6 100644
--- a/drivers/platform/tegra/tegra21_clocks.c
+++ b/drivers/platform/tegra/tegra21_clocks.c
@@ -1061,7 +1061,7 @@ static struct clk_ops tegra_super_ops = {
  */
 static void tegra21_cpu_clk_init(struct clk *c)
 {
-	c->state = (!is_lp_cluster() == (c->u.cpu.mode == MODE_G)) ? ON : OFF;
+	c->state = ((!is_lp_cluster()) == (c->u.cpu.mode == MODE_G)) ? ON : OFF;
 }
 
 static int tegra21_cpu_clk_enable(struct clk *c)

  1. Download and unzip the relevant cross-compilation toolchain for your kernel and compile it, and make the kernel_supplements tarball (e.g. for R24.2.1 https://developer.ridgerun.com/wiki/index.php?title=Compiling_Jetson_TX1_source_code_-_Jetpack_3.0)

  1. Copy the compiled 3.10.96+ kernel and modules tarball for use in the upgrade.zip.

  1. Copy the extlinux.conf from an old TX1, and add a secondary entry pointing at the backup kernel, and include this in the upgrade.zip, e.g.
TIMEOUT 30
DEFAULT primary

MENU TITLE p2371-2180 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      FDT /boot/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
      APPEND fbcon=map:0 console=tty0 console=ttyS0,115200n8 androidboot.modem=none androidboot.serialno=P2180A00P00940c003fd androidboot.security=non-secure tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=${lp0_vec} nvdumper_reserved=${nvdumper_reserved} core_edp_mv=1125 core_edp_ma=4000 gpt android.kerneltype=normal androidboot.touch_vendor_id=0 androidboot.touch_panel_id=63 androidboot.touch_feature=0 androidboot.bootreason=pmc:software_reset,pmic:0x0 net.ifnames=0 root=/dev/mmcblk0p1 rw rootwait

LABEL backup
    MENU LABEL backup kernel
    LINUX /boot/Image.backup
    INITRD /boot/initrd
    FDT /boot/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
    APPEND fbcon=map:0 console=tty0 console=ttyS0,115200n8 androidboot.modem=none androidboot.serialno=P2180A00P00940c003fd androidboot.security=non-secure tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=${lp0_vec} nvdumper_reserved=${nvdumper_reserved} core_edp_mv=1125 core_edp_ma=4000 gpt android.kerneltype=normal androidboot.touch_vendor_id=0 androidboot.touch_panel_id=63 androidboot.touch_feature=0 androidboot.bootreason=pmc:software_reset,pmic:0x0 net.ifnames=0 root=/dev/mmcblk0p1 rw rootwait


  1. Flash a fresh TX1 using JetPack 4.4. Be sure to use exactly the same APP partition size as you used in the previous versions in the field. New versions of JetPack include a recovery rootfs partition (RECROOTFS), which may end up adding up to more than the available disk space.
    For example, if you used ./flash.sh -S 14580MiB, you’ll need to do something like this to make the partitions fit:
sed -i "s|<size> RECROOTFSSIZE </size>|<size> 167772160 </size>|" /Linux_for_Tegra/bootloader/t210ref/cfg/gnu_linux_tegraboot_emmc_full.xml

  1. On the newly flashed TX1, clone the GPT partition table, and all of the partition images, including the 2 bootloader partitions (except the APP partition), and also the /boot folder. Make a note of the partition alignment of partition 2, as we’ll need to check this is the same as on the old TX1s in the field.
mkdir -p /home/ubuntu/upgrade/
# Take a backup of the GPT table from the newly flashed tegra
sudo sgdisk --backup=/home/ubuntu/upgrade/gpt_table_new.bin /dev/mmcblk0

# Clone all the partitions except APP
sudo dd if=/dev/mmcblk0p2 of=/home/ubuntu/upgrade/mmcblk0p2.img
sudo dd if=/dev/mmcblk0p3 of=/home/ubuntu/upgrade/mmcblk0p3.img
sudo dd if=/dev/mmcblk0p4 of=/home/ubuntu/upgrade/mmcblk0p4.img
sudo dd if=/dev/mmcblk0p5 of=/home/ubuntu/upgrade/mmcblk0p5.img
sudo dd if=/dev/mmcblk0p6 of=/home/ubuntu/upgrade/mmcblk0p6.img
sudo dd if=/dev/mmcblk0p7 of=/home/ubuntu/upgrade/mmcblk0p7.img
sudo dd if=/dev/mmcblk0p8 of=/home/ubuntu/upgrade/mmcblk0p8.img
sudo dd if=/dev/mmcblk0p9 of=/home/ubuntu/upgrade/mmcblk0p9.img
sudo dd if=/dev/mmcblk0p10 of=/home/ubuntu/upgrade/mmcblk0p10.img
sudo dd if=/dev/mmcblk0p11 of=/home/ubuntu/upgrade/mmcblk0p11.img
sudo dd if=/dev/mmcblk0p12 of=/home/ubuntu/upgrade/mmcblk0p12.img
sudo dd if=/dev/mmcblk0p13 of=/home/ubuntu/upgrade/mmcblk0p13.img
sudo dd if=/dev/mmcblk0p14 of=/home/ubuntu/upgrade/mmcblk0p14.img
sudo dd if=/dev/mmcblk0p15 of=/home/ubuntu/upgrade/mmcblk0p15.img
sudo dd if=/dev/mmcblk0p16 of=/home/ubuntu/upgrade/mmcblk0p16.img
sudo dd if=/dev/mmcblk0p17 of=/home/ubuntu/upgrade/mmcblk0p17.img
sudo dd if=/dev/mmcblk0p18 of=/home/ubuntu/upgrade/mmcblk0p18.img
sudo dd if=/dev/mmcblk0p19 of=/home/ubuntu/upgrade/mmcblk0p19.img
sudo dd if=/dev/mmcblk0p20 of=/home/ubuntu/upgrade/mmcblk0p20.img
sudo dd if=/dev/mmcblk0p21 of=/home/ubuntu/upgrade/mmcblk0p21.img
sudo dd if=/dev/mmcblk0p22 of=/home/ubuntu/upgrade/mmcblk0p22.img
sudo dd if=/dev/mmcblk0boot0 of=/home/ubuntu/upgrade/mmcblk0boot0.img
sudo dd if=/dev/mmcblk0boot1 of=/home/ubuntu/upgrade/mmcblk0boot1.img
sudo cp -r /boot /home/ubuntu/upgrade/

# Make a note of the start of partition 2 for later
sudo parted /dev/mmcblk0 unit B print

  1. Get the checksums of the relevant partitions and files using:
find /dev -type b -name "mmcblk0*" -not -name "*rpmb" -not -name "mmcblk0" -not -name "mmcblk0p1" | xargs sudo sha256sum
find /boot -type f | xargs sha256sum

Save these for use in the upgrade.zip so we can use sha256sum --check to verify the upgrade later.


  1. Install apt-clone and make an apt clone of the current versions of packages on the new TX1 using
apt-clone clone jetpack-4-4-1-tx1

Copy the resulting jetpack-4-4-tx1.apt-clone.tar.gz for use in the upgrade.zip


  1. Make a file called nvidia-l4t-apt-source.list containing:
deb https://repo.download.nvidia.com/jetson/common r32.4 main
deb https://repo.download.nvidia.com/jetson/t210 r32.4 main

  1. Copy the l4t_payload_updater_t210 from /usr/sbin/ for use in the upgrade.zip.

If adapting this process for TX2, you want /usr/sbin/nv_update_engine instead. You’ll also want to unarchive the bootloader deb package to retrieve the postinst script so we can use it later:

# For TX2 only!
ar -x Linux_for_Tegra/bootloader/nvidia-l4t-bootloader_32.4.4-20201016123640_arm64.deb
tar -xvf control.tar.gz
# Now copy control/postinst for use in upgrade.zip

  1. Copy the jetson-ota-public.asc from /etc/apt/trusted.gpg.d/jetson-ota-public.asc for use in the upgrade.zip

  1. On an 18.04 x86 machine running JetPack 4.4, copy the kernel deb packages into the upgrade.zip (or provide your own packages if patching DTBs/kernels for your board):
Linux_for_Tegra/kernel/nvidia-l4t-kernel_4.9.140-tegra-32.4.4-20201016124427_arm64.deb
Linux_for_Tegra/kernel/nvidia-l4t-kernel-dtbs_4.9.140-tegra-32.4.4-20201016124427_arm64.deb
Linux_for_Tegra/kernel/nvidia-l4t-kernel-headers_4.9.140-tegra-32.4.4-20201016124427_arm64.deb

  1. Generate a bootloader update payload for your board and copy it from Linux_for_Tegra/bootloader/payloads_t21x/bl_update_payload into the upgrade.zip. This step isn’t necessary if you didn’t use a custom APP partition size:
sed -i '/partitions_jetson_nano_qspi_sd[@]/d' ./bootloader/l4t_bup_gen.func   # (to skip generating nano payload, which can fail)
sudo ./flash.sh -S 14580MiB --bup jetson-tx1 mmcblk0p1

If adapting for TX2, the output will be in Linux_for_Tegra/bootloader/payloads_t18x/bl_update_payload


Running the upgrade

After rolling all the above into upgrade.zip, these are the steps to actually run an upgrade on an old TX1:

  1. Download and unzip and checksum everything in the upgrade.zip

  1. Backup the old kernel, and copy the patched 3.10.96+ kernel and modules, and updated extlinux.conf to the right places, e.g.:
sudo cp /boot/Image /boot/Image.backup
sudo cp /home/ubuntu/upgrade/modules/lib/modules /lib/ -r
sudo cp /home/ubuntu/upgrade/extlinux.conf /boot/extlinux/extlinux.conf
sudo cp /boot/Image /boot/Image.backup
sudo cp /home/ubuntu/upgrade/Image.3.0 /boot/Image
echo "a9a7d3d21db9e3f9712e6b3ecb67e155dd2d37647344a9f1e3b626d3cfb45490  /boot/Image" | sha256sum

  1. Reboot, and make sure /dev/mmcblk0boot0 and /dev/mmcblk0boot1 are visible under lsblk

  1. Verify that your partition alignment is exactly what you expect using:
sudo parted /dev/mmcblk0 unit B print

Specifically, the start of partition 2 MUST be aligned with that of the new JetPack 4.4 TX1.

If it isn’t you should abort here!


  1. Here’s where it gets interesting. The Tegra bootloader updater expects a file at /etc/nv_boot_control.conf which contains information about the hardware revision. Normally this hardware information is read from the CVM EEPROM during flash, and is copied onto the rootfs before flashing the APP partition. In our case, we need to create it after the fact by reading the CVM EEPROM ourselves from the running tegra, as below:

# Read bytes from 0x14-0x29 of the CVM EEPROM (this is normally read into Linux_for_Tegra/bootloader/cvm.bin by the flashing host during flash time) and extract the TNSPEC
# See https://developer.nvidia.com/embedded/dlc/tx1-tx2-module-eeprom-layout

# A function to read multiple bytes using hex addresses from the tegra CVM EEPROM on I2C bus 2, address 50.  
# TX2s have this EEPROM on bus 7 instead, so amend the read_eeprom function and expected values accordingly for TX2 (e.g. BOARD_ID will be 3310).
# e.g. read_eeprom 14 15 16
read_eeprom() {
  for var in "$@"
  do
    sudo i2cget -y 2 0x50 0x$var b | xxd -r;
  done
}

SIX_NINE_NINE=$(read_eeprom 14 15 16)

if [ $SIX_NINE_NINE != "699" ]; then
  echo "expected EEPROM read on I2C bus 2 from 0x14-0x16 to be '699', got $SIX_NINE_NINE";
  exit 1;
fi

EIGHT=$(read_eeprom 18)

if [ $EIGHT != "8" ]; then
  echo "expected EEPROM read on I2C bus 2 from 0x18 to be '8', got $EIGHT";
  exit 1;
fi

BOARD_ID=$(read_eeprom 19 1a 1b 1c)

if [ $BOARD_ID != "2180" ]; then
  echo "expected BOARD_ID read on I2C bus 2 from 0x19-0x1c to be '2180', got $BOARD_ID";
  exit 1;
fi

SKU=$(read_eeprom 1e 1f 20 21)

if [ $SKU != "1000" ]; then
  echo "expected SKU read on I2C bus 2 from 0x1e-0x21 to be '1000', got $SKU";
  exit 1;
fi

BOARD_VERSION_FAB=$(read_eeprom 23 24 25)

# TODO - test that FAB is known to us and compatible with this bootloader BCT (e.g. 400/401)


BOARD_REV=$(read_eeprom 27 28 29)

# TODO - test that BOARDREV is known to us and compatible with this bootloader BCT

# Note, the TNSPEC depends on the exact hardware revision with this pattern: BOARD_ID-FAB-BOARDSKU-BOARDREV-NV_PRODUCTION-CHIP_REV-BOARD_NAME-ROOTFS_DEV
sudo bash -c "cat <<EOT > /etc/nv_boot_control.conf
TNSPEC ${BOARD_ID}-${BOARD_VERSION_FAB}-${SKU}-${BOARD_REV}-1-0-jetson-tx1-mmcblk0p1
COMPATIBLE_SPEC ${BOARD_ID}----1--jetson-tx1-mmcblk0p1
TEGRA_CHIPID 0x21
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1
EOT"

If adapting these steps for TX2, be sure to replace the I2C bus to 7, and the BOARD_ID (3310) BOARD_NAME (jetson-tx2) and TEGRA_CHIPID (0x18) accordingly.


  1. Take a backup of all partitions other than APP, and of the existing GPT partition table:

mkdir -p /home/ubuntu/upgrade/backup

cd /home/ubuntu/upgrade/backup

sudo dd if=/dev/mmcblk0p2 of=/home/ubuntu/upgrade/backup/mmcblk0p2.img
sudo dd if=/dev/mmcblk0p3 of=/home/ubuntu/upgrade/backup/mmcblk0p3.img
sudo dd if=/dev/mmcblk0p4 of=/home/ubuntu/upgrade/backup/mmcblk0p4.img
sudo dd if=/dev/mmcblk0p5 of=/home/ubuntu/upgrade/backup/mmcblk0p5.img
sudo dd if=/dev/mmcblk0p6 of=/home/ubuntu/upgrade/backup/mmcblk0p6.img
sudo dd if=/dev/mmcblk0p7 of=/home/ubuntu/upgrade/backup/mmcblk0p7.img
sudo dd if=/dev/mmcblk0p8 of=/home/ubuntu/upgrade/backup/mmcblk0p8.img
sudo dd if=/dev/mmcblk0p9 of=/home/ubuntu/upgrade/backup/mmcblk0p9.img
sudo dd if=/dev/mmcblk0p10 of=/home/ubuntu/upgrade/backup/mmcblk0p10.img
sudo dd if=/dev/mmcblk0p11 of=/home/ubuntu/upgrade/backup/mmcblk0p11.img
sudo dd if=/dev/mmcblk0p12 of=/home/ubuntu/upgrade/backup/mmcblk0p12.img
sudo dd if=/dev/mmcblk0p13 of=/home/ubuntu/upgrade/backup/mmcblk0p13.img
sudo dd if=/dev/mmcblk0p14 of=/home/ubuntu/upgrade/backup/mmcblk0p14.img
sudo dd if=/dev/mmcblk0p15 of=/home/ubuntu/upgrade/backup/mmcblk0p15.img
sudo dd if=/dev/mmcblk0p16 of=/home/ubuntu/upgrade/backup/mmcblk0p16.img
sudo dd if=/dev/mmcblk0p17 of=/home/ubuntu/upgrade/backup/mmcblk0p17.img
sudo dd if=/dev/mmcblk0p18 of=/home/ubuntu/upgrade/backup/mmcblk0p18.img
sudo dd if=/dev/mmcblk0boot0 of=/home/ubuntu/upgrade/backup/mmcblk0boot0.img
sudo dd if=/dev/mmcblk0boot1 of=/home/ubuntu/upgrade/backup/mmcblk0boot1.img

# Take a backup of the whole GPT table
sudo sgdisk --backup=/home/ubuntu/upgrade/backup/gpt_table.bin /dev/mmcblk0

These could be used in case we need to roll back.

Obviously if adapting for TX2, make sure to copy all the extra partitions.


  1. If you have a non-standard partition layout (e.g. if you used ./flash.sh -S 14580MiB), standard bootloader update payloads will contain incorrect partition addresses in the BCT. To avoid the standard l4t bootloader updater deb package writing its own copy of the partition table, we can disable the boot firmware update during postinst by touching this file: /opt/nvidia/l4t-packages/.nv-l4t-disable-boot-fw-update-in-preinstall
sudo mkdir -p /opt/nvidia/l4t-packages
sudo touch /opt/nvidia/l4t-packages/.nv-l4t-disable-boot-fw-update-in-preinstall

We can then (later) flash the bootloader using our custom bl_update_payload we created earlier


  1. (Optional) To stop us being prompted for what to do with package configuration files during upgrade - always choose the new version
sudo bash -c 'cat <<EOF > /etc/apt/apt.conf.d/local
Dpkg::Options {
   "--force-confnew";
}
EOF'

  1. Free up some space, and uninstall any old versions of packages which you don’t want after the upgrade:
sudo apt install -y apt-transport-https
sudo apt-get remove -y nv-gie-repo-ubuntu1604-6-rc-cuda8.0 cuda-repo-l4t-8-0-local cuda-toolkit-8-0 cuda-cublas-8-0 cuda-cudart-8-0 cuda-cudart-dev-8-0 cuda-cufft-8-0 cuda-cufft-dev-8-0 cuda-driver-dev-8-0 cuda-license-8-0 cuda-npp-8-0 cuda-npp-dev-8-0 libboost1.58-dev rhythmbox ubuntu-session libreoffice* firefox linux-headers*

sudo apt-get autoremove -y

  1. Install apt-clone
sudo apt-get update
sudo apt-get install -y apt-clone

  1. Update apt sources from xenial to bionic, and add the l4t apt key and sources:
sudo sed -i "s|xenial|bionic|" /etc/apt/sources.list
sudo apt-key add /home/ubuntu/upgrade/4.4/jetson-ota-public.key
sudo cp /home/ubuntu/upgrade/4.4/nvidia-l4t-apt-source.list /etc/apt/sources.list.d/

sudo rm -f /etc/apt/sources.list.d/nv-gie-6-rc-cuda8.0.list
sudo rm -f /etc/apt/sources.list.d/cuda-8-0-local.list
sudo rm -f /etc/apt/sources.list.d/libopencv4tegra-repo.list

Everything before this point was reversible, but beyond here is the point of no return.

  1. Download all the files needed for a dist-upgrade:
sudo apt-get update && sudo apt-get -y dist-upgrade --download-only || sudo dpkg --force-confnew --configure -a
sudo apt-get -f install -y

The new 4.9.120 kernel depends on a newer version of udev to provide firmware to the kernel on demand. If you were to apply the upgrade in a different order (e.g. upgrade the GPT, partition images, kernel, DTBs and bootloader first, then do the dist-upgrade), this would still work (and you could safely reboot into the 4.9.120 kernel while still on 16.04), but various hardware (including WiFi) wouldn’t work due to the old version of udev running on 16.04, and old firmware inside /lib.


Warning - rebooting anywhere between here and the completion of step 19 will mean you need to reflash.

It is possible to skip straight to steps 16-19 after 13, and complete steps 14 and 15 after successfully updating the bootloader/kernel/dtbs and rebooting, but not all hardware will work due to a different version of udev and firmware on 16.04, which won’t work with a 4.9.120 kernel.
  1. Install the nvidia-l4t-kernel, modules, DTBs and headers:
# Move the old kernel/DTBs into a backup folder first
sudo mkdir -p /boot/backups
sudo mv /boot/* /boot/backups || true

sudo apt install -y /home/ubuntu/upgrade/4.4/nvidia-l4t-kernel_4.9.140-tegra-32.4.4-20201016124427_arm64.deb
sudo apt install -y /home/ubuntu/upgrade/4.4/nvidia-l4t-kernel-dtbs_4.9.140-tegra-32.4.4-20201016124427_arm64.deb
sudo apt install -y /home/ubuntu/upgrade/4.4/nvidia-l4t-kernel-headers_4.9.140-tegra-32.4.4-20201016124427_arm64.deb

  1. Start apt-clone restore. The restore might clobber the nvidia-l4t-apt-source.list, so put it back. This might fail due to postinstall dependency cyles, so keep trying these steps repeatedly until you get no more errors.
sudo apt-clone restore /home/ubuntu/upgrade/4.4/jetpack-4-4-tx1.apt-clone.tar.gz || \
sudo dpkg --force-confnew --configure -a
sudo apt-get -f install -y
sudo cp /home/ubuntu/upgrade/4.4/nvidia-l4t-apt-source.list /etc/apt/sources.list.d/

We need to prevent the nvidia-l4t-bootloader package from updating automatically in future and and flashing the bootloader partition with a BCT which contains an incorrect mmcblk0 partition table (this only applies if we have a custom APP partition size), so we hold that package version so that future bootloader upgrades can be done carefully/manually. If you allow the nvidia-l4t-bootloader package to update automatically, the Tegra would get as far as NVTBoot -> CBoot but then it would look for u-boot in the LNX partition (which is in mmcblk0p5) at the wrong partition start address and fail to boot any further.

*Nvidia should probably release an update to the bootloader updater to do a check in preinst that the BUP’s BCT copy of the partition table matches the actual mmcblk0 partition table, as this is easy to check, and many people have custom partition sizes. It should also be easy enough to amend the BCT partition table copy based on the actual address of the LNX partition, allowing people to remotely update their bootloader safely, even with custom partitions.

# Freeze the version of the bootloader to prevent it flashing with a new payload which has the wrong partition table on it. 
# Bootloader updates will need a custom payload, generated using sudo ./flash.sh -S 14580MiB --bup jetson-tx1 mmcblk0p1
sudo apt-mark hold nvidia-l4t-bootloader
sudo touch /opt/nvidia/l4t-packages/.nv-l4t-disable-boot-fw-update-in-preinstall

  1. By this point, systemd and dbus might not be in good shape, so you might need to mess around with /etc/resolv.conf to get DNS working
sudo rm -f /etc/resolv.conf
sudo bash -c "cat <<EOF > /etc/resolv.conf
nameserver 8.8.8.8
EOF"

# Do what you need with internet, once you're done, let systemd's stub resolver take over before rebooting
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Warning - This operation will be destructive and irreversible if you didn’t verify the start address of partition 2 against the Jetpack 4.4 version. Otherwise, it is theoretically reversible.

  1. Overwrite the current GPT layout by restoring from the backup we took on the JetPack 4.4 TX1, then run partprobe to refresh the kernel’s view of the partitions:
sudo sgdisk --load-backup=/home/ubuntu/upgrade/4.4/partitions/gpt_table_new.bin /dev/mmcblk0
sudo partprobe -s /dev/mmcblk0

  1. Overwrite all the partitions with the new versions:
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p2.img of=/dev/mmcblk0p2
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p3.img of=/dev/mmcblk0p3
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p4.img of=/dev/mmcblk0p4
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p5.img of=/dev/mmcblk0p5
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p6.img of=/dev/mmcblk0p6
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p7.img of=/dev/mmcblk0p7
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p8.img of=/dev/mmcblk0p8
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p9.img of=/dev/mmcblk0p9
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p10.img of=/dev/mmcblk0p10
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p11.img of=/dev/mmcblk0p11
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p12.img of=/dev/mmcblk0p12
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p13.img of=/dev/mmcblk0p13
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p14.img of=/dev/mmcblk0p14
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p15.img of=/dev/mmcblk0p15
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p16.img of=/dev/mmcblk0p16
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p17.img of=/dev/mmcblk0p17
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p18.img of=/dev/mmcblk0p18
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p19.img of=/dev/mmcblk0p19
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p20.img of=/dev/mmcblk0p20
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p21.img of=/dev/mmcblk0p21
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0p22.img of=/dev/mmcblk0p22

# Make the bootloader partitions writeable
sudo /bin/bash -c 'echo 0 > /sys/block/mmcblk0boot0/force_ro'
sudo /bin/bash -c 'echo 0 > /sys/block/mmcblk0boot1/force_ro'

sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0boot0.img of=/dev/mmcblk0boot0
sudo dd if=/home/ubuntu/upgrade/4.4/partitions/mmcblk0boot1.img of=/dev/mmcblk0boot1

# Return the bootloader partitions to read only
sudo /bin/bash -c 'echo 1 > /sys/block/mmcblk0boot0/force_ro'
sudo /bin/bash -c 'echo 1 > /sys/block/mmcblk0boot1/force_ro'


  1. Checksum the all the partitions against the checksums collected from the 4.4 version. If they don’t match, you can try to rollback the GPT and partitions to their previous contents using sgdisk --load-backup and dd.
# This was created by running the below on a clean 4.4 flash:
# find /dev -type b -name "mmcblk0*" -not -name "*rpmb" -not -name "mmcblk0" -not -name "mmcblk0p1" | xargs sudo sha256sum

cat <<EOF | sudo sha256sum --check
ce8b6c162f4e898b1ea252f1100aa4feaa67ff8e99ea37e609e4357448757c35  /dev/mmcblk0boot0
bd518dd98c30d31dff0f0742c594a99718a1e68cac9fe15698dc2246cfaec1fb  /dev/mmcblk0boot1
b5c5c742ef6e4dbe6225f8438fa7f00796de36367dda75428eeedb774308fb85  /dev/mmcblk0p22
61b5d2e238243a70dd9e9ad76225379515134a2531f374f960f5c6b5cf42519d  /dev/mmcblk0p21
5647f05ec18958947d32874eeb788fa396a05d0bab7c1b71f112ceb7e9b31eee  /dev/mmcblk0p20
b69dae56a14d1a8314ed40664c4033ea0a550eea2673e04df42a66ac6b9faf2c  /dev/mmcblk0p19
b69dae56a14d1a8314ed40664c4033ea0a550eea2673e04df42a66ac6b9faf2c  /dev/mmcblk0p18
5647f05ec18958947d32874eeb788fa396a05d0bab7c1b71f112ceb7e9b31eee  /dev/mmcblk0p17
3b6a07d0d404fab4e23b6d34bc6696a6a312dd92821332385e5af7c01c421351  /dev/mmcblk0p16
8a39d2abd3999ab73c34db2476849cddf303ce389b35826850f9a700589b4a90  /dev/mmcblk0p15
8a39d2abd3999ab73c34db2476849cddf303ce389b35826850f9a700589b4a90  /dev/mmcblk0p14
07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541  /dev/mmcblk0p13
bf25a5db8ce4f55e99bd25447242b749a39c32108083b78cf3185cd4d1d0a893  /dev/mmcblk0p12
29f6007260282cf7075a17b7ca43906e094c9b7ace7faefa0a62fa0fb5643253  /dev/mmcblk0p11
0ae989f41e9f423f8e90b4d35e878d6b63d3bdb71ba3ab425a3eb659045e4ef0  /dev/mmcblk0p10
051337e11e77b3a9b4660f08da0b1e098608ad46021446b965fde4c129e7126d  /dev/mmcblk0p9
15ff6656c8d14473df20dc9794030243e2c8d68435be9a36491da7fc7af9c0e5  /dev/mmcblk0p8
083a7c550d169515c121574ac117634c1a75371e662c72de2106dd42e43ebe2f  /dev/mmcblk0p7
31e05093d93ec186494ab4048ffc0a90688dbdcb2e4495b529bd5874a8f47965  /dev/mmcblk0p6
051337e11e77b3a9b4660f08da0b1e098608ad46021446b965fde4c129e7126d  /dev/mmcblk0p5
15ff6656c8d14473df20dc9794030243e2c8d68435be9a36491da7fc7af9c0e5  /dev/mmcblk0p4
083a7c550d169515c121574ac117634c1a75371e662c72de2106dd42e43ebe2f  /dev/mmcblk0p3
31e05093d93ec186494ab4048ffc0a90688dbdcb2e4495b529bd5874a8f47965  /dev/mmcblk0p2
EOF

  1. If dbus isn’t responsive, reboots might not work so:
# Force a hard sysrq reboot because dbus will likely be down/unresponsive to regular reboots
sudo bash -c "sleep 10 && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && echo s > /proc/sysrq-trigger && echo b > /proc/sysrq-trigger"&
sudo reboot

  1. Pray that your TX1/TX2 comes back alive

  1. If it did, run the bootloader update using our custom bl_update_payload. This step should be a no-op, and is mainly to prove that remote bootloader updates using BUPs will be possible from now on.
# Apply the custom bootloader update - this should leave most bootloader partitions untouched relative to the image we dd'd earlier, but it should do a checksum and validation of everything to be safe
sudo /home/ubuntu/upgrade/4.4/l4t_payload_updater_t210 /home/ubuntu/upgrade/4.4/bl_update_payload

If adapting this process for TX2, it’s safer to let the deb package’s postinst script run the bootloader update, since there are differences in the SDRAM timings placed in the BCT depending on the exact hardware revision (i.e. Micron vs Samsung memory):

# TX2 only!
sudo cp /home/ubuntu/upgrade/4.4/nv_update_engine /usr/sbin/nv_update_engine
sudo mkdir -p /opt/ota_package/t18x
sudo cp /home/ubuntu/upgrade/4.4/bl_update_payload /opt/ota_package/t18x/bl_update_payload

# This postinst script came from unarchiving the nvidia-l4t-bootloader deb file, and untar'ing the control.tar.gz within
sudo /home/ubuntu/upgrade/4.4/postinst 

cat /opt/ota_package/bl_update_payload.log

  1. Remove any old packages and free up space, and return the dkpg configuration to normal by removing the apt config we added earlier
sudo rm /etc/apt/apt.conf.d/local

sudo apt-get remove -y libreoffice*
sudo apt-get remove -y firefox
sudo apt-get remove -y linux-headers*
sudo apt-get remove -y fonts-noto-cjk fonts-noto-color-emoji fonts-nanum 
sudo apt-get remove -y libjansson-dev cmake build-essential libjpeg8-dev libtiff5-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev gfortran  libatlas-base-dev
sudo apt-get autoremove -y
sudo apt-get autoclean -y
sudo rm -rf /usr/lib/chromium-browser

sudo apt-get clean -y

# Remove any .so files which came from R24.2.1, since they'll be obsolete now
sudo find / -name "*.24.2.1*" | xargs sudo rm -rf

sudo rm -rf /home/ubuntu/upgrade/4.4

sudo journalctl --vacuum-size=200M

# Clean up old backups if you're sure everything is ok
sudo rm -rf /boot/backups

# Remove kernel modules for old kernels
sudo rm -rf /lib/modules/3.10.96-tegra
sudo rm -rf /lib/modules/3.10.96+
sudo rm -rf /lib/firmware/amdgpu
sudo rm -rf /usr/src/linux-headers-3.10.96-tegra
sudo rm -rf /usr/src/nvidia

find /usr/share/locale -type d -not -wholename "/usr/share/locale/en*" | xargs sudo rm -rf
sudo locale-gen --purge en_US

sudo rm -rf /usr/lib/chromium-browser
sudo rm -rf /var/lib/apt/lists/*
sudo rm -rf /home/ubuntu/upgrade

Hopefully people find this helpful.

2 Likes

I have not tried this, but I must say I am impressed with the details. There is probably a lot to be learned by studying this.

Hi,
Thanks for the sharing. The deviation between JP3.0 and JP4.4.1 is huge and it is really challenging and risky. Thanks for sharing the method to help other users.

Hi eh-steve,

Thanks a lot for sharing this upgrade method you have worked out. It was obviously a lot of work to figure out and I’m happy to learn it is possible (even if risky). I agree with linuxdev that there is a lot to learn from the info you posted. Congratulations on making it work!

Thanks,

Chris Richardson