Hi,
I have been updating NVME drive internally attached to Jetson via USB-stick as a remote update mechanism via USB.
Steps which I followed was:
- Prepared the internal and external images for NVME and bootloader
#internal image creation
sudo ADDITIONAL_DTB_OVERLAY_OPT="BootOrderNvme.dtbo" BOARDID=3668 FAB=301 BOARDSKU=0001 BOARDREV=E.0 ./tools/kernel_flash/l4t_initrd_flash.sh -u $ROOT_DIR/keys/jetson.pem -v $ROOT_DIR/keys/sbk_key.txt -p "-c bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml" --showlogs --no-flash jetson-xavier-nx-devkit-emmc internal
#external image creation
sudo ADDITIONAL_DTB_OVERLAY_OPT="BootOrderNvme.dtbo" ROOTFS_ENC=1 BOARDID=3668 FAB=301 BOARDSKU=0001 BOARDREV=E.0 ./tools/kernel_flash/l4t_initrd_flash.sh -u $ROOT_DIR/keys/jetson.pem -v $ROOT_DIR/keys/sd/sbk_key.txt -i ./ekb.key ``-p "--generic-passphrase"`` --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t194_nvme_rootfs_enc.xml -S 40GiB --no-flash --append --direct nvme0n1 --showlogs jetson-xavier-nx-devkit-emmc external
- The above step creates internal and external images within Linux_for_Tegra/ folder. I created a backup of this folder without all raw images so that the size can be reduced.
- Create a minimal rootfs to be booted from USB drive.
- Untar the Linux_for_Tegra folder in this new minimal rootfs. The idea is after untar, I will have all the flash images and prerequistes to flash the NVME from bootable USB stick.
- Create a usb sda1 bootable image .img using loop device.
# create file for usb-image
sudo dd if=/dev/zero of=image.iso bs=1M count=64000 status=progress
# mount usb-image on loop device
loop_device_path=$(sudo losetup -f)
sudo losetup $loop_device_path image.iso
loop_device_path_name="${loop_device_path##*/}"
#Format Loop device
echo "Formatting $loop_device_path with ext4 filesystem..."
sudo mkfs.ext4 -F $loop_device_path
pushd $ROOT_DIR/Linux_for_Tegra
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --direct $loop_device_path_name --external-device sda1 -c ./tools/kernel_flash/flash_l4t_external.xml -S 40GiB jetson-xavier-nx-devkit-emmc external
popd
-
At the end of this step an image.iso has been generated which can be flashed to a USB-stick and then the jetson can be booted up from USB stick.
-
At boot-up from USB stick, there is a startup service which updates the internally connected Nvme drive
pushd /Linux_for_Tegra
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml -S 40GiB --direct nvme0n1 --showlogs jetson-xavier-nx-devkit-emmc external
popd
The issue is: this method works until bootloader or bsp doesn’t need to be changed. When a bsp change is required, this fails with “Kernel panic” as the bootloader on the QSPI-NOR memory is old and it cannot load new kernel modules.
How can I also update the bootloader with this approach along with NVME ?
I tried on the bootable USB to update it by updating internal image but it cannot be done. I thought in the earlier step, I used --append option to append the internal qspi image with the external, this should work.
pushd /Linux_for_Tegra
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only jetson-xavier-nx-devkit-emmc internal
[sudo] password for jetson:
No devices to flash
popd
Is it even possible to update bootloader when the jetson is booted up from USB ? I want the bootloader + NVME to be new images from the next reboot onwards when I remove the USB, it should bootup with new bootloader + new image on the NVME.
Thanks !
Hi,
Some questions to confirm:
Do you use the dev kit or the custom carrier board?
Thanks
I use the custom carrier board.
I am trying to update from R35.4 BSP to R35.6.0.
Hi,
In your situation, please refer to the Software Packages and the Update Mechanism — NVIDIA Jetson Linux Developer Guide 1 documentation to do the image-based ota for your custom board.
Thanks
Use case: Update bootloader only
- Generated OTA Payload for NVME
export BASE_BSP=/home/oldbsp/Linux_for_Tegra #R35.4.1
export TARGET_BSP=/home/newbsp/Linux_for_Tegra #R35.6.0
export BASE_BSP_VERSION=R35-4
export TARGET_JETSON_DEVICE=jetson-xavier-nx-devkit-emmc
#Generated OTA PAYLOAD successfully for bootloader update only !
sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -b -u ../keys/jetson.pem -v ../keys/sbk_key.txt --external-device nvme0n1 -S 40GiB ${TARGET_JETSON_DEVICE} ${BASE_BSP_VERSION}
I used -u -v PKC key & SBK key for signing as my jetson board is fused with these keys. I also have disk encryption enabled with generic passphrase on nvme. Do I have to use ROOTFS_ENC=1 and -i ekb.key in this ota payload generation ?
- After step 1, “Linux_for_Tegra/bootloader/jetson-xavier-nx-devkit-emmc/ota_payload_package.tar.gz” is successfully generated.
- Copied this payload tar.gz to the jetson device on NVME /boot/ota partion and applied it via ./nv_ota_start.sh /boot/ota/ota_payload_package.tar.gz
nvme0n1 259:0 0 119.2G 0 disk
├─nvme0n1p1 259:1 0 400M 0 part /boot
├─nvme0n1p2 259:2 0 39.6G 0 part
│ └─crypt_root 252:0 0 39.6G 0 crypt /
├─nvme0n1p3 259:3 0 128M 0 part
├─nvme0n1p4 259:4 0 768K 0 part
├─nvme0n1p5 259:5 0 31.6M 0 part
├─nvme0n1p6 259:6 0 128M 0 part
├─nvme0n1p7 259:7 0 768K 0 part
├─nvme0n1p8 259:8 0 31.6M 0 part
├─nvme0n1p9 259:9 0 80M 0 part
├─nvme0n1p10 259:10 0 512K 0 part
├─nvme0n1p11 259:11 0 100M 0 part
├─nvme0n1p12 259:12 0 64M 0 part
├─nvme0n1p13 259:13 0 80M 0 part
├─nvme0n1p14 259:14 0 512K 0 part
├─nvme0n1p15 259:15 0 64M 0 part
└─nvme0n1p16 259:16 0 78.6G 0 part
└─crypt_UDA 252:1 0 78.5G 0 crypt /mnt/crypt_UDA```
root@JETSONE1:/boot# tree ota
ota
├── ota_payload_package.tar.gz
└── ota-tools
├── Linux_for_Tegra
│ ├── bootloader
│ │ └── t186ref
│ │ ├── BCT
│ │ │ └── tegra186-mb1-bct-misc-si-l4t-r32i.cfg
│ │ └── cfg
│ │ ├── flash_l4t_t194_spi_emmc_jaxi_R32x_R35i.xml
│ │ ├── flash_l4t_t194_spi_emmc_jaxi_R35A_R35i.xml
│ │ ├── flash_l4t_t194_spi_emmc_jaxi_rootfs_ab_R32x_R35i.xml
│ │ ├── flash_l4t_t194_spi_emmc_jaxi_rootfs_ab_R35A_R35i.xml
│ │ ├── flash_l4t_t194_spi_emmc_p3668_R32i.xml
│ │ ├── flash_l4t_t194_spi_emmc_p3668_R32x_R35i.xml
│ │ ├── flash_l4t_t194_spi_emmc_p3668_R35A_R35i.xml
│ │ ├── flash_l4t_t194_spi_emmc_p3668_rootfs_ab_R32i.xml
│ │ ├── flash_l4t_t194_spi_emmc_p3668_rootfs_ab_R32x_R35i.xml
│ │ ├── flash_l4t_t194_spi_emmc_p3668_rootfs_ab_R35A_R35i.xml
│ │ ├── flash_t194_sdmmc_R32x_R35i.xml
│ │ ├── flash_t194_sdmmc_R35A_R35i.xml
│ │ ├── flash_t194_sdmmc_rootfs_ab_R32x_R35i.xml
│ │ └── flash_t194_sdmmc_rootfs_ab_R35A_R35i.xml
│ ├── jaxi-R32x-R35i.conf -> p2822-0000+p2888-0008-R32x-R35i.conf
│ ├── jaxi-R35A-R35i.conf -> p2822-0000+p2888-0008-R35A-R35i.conf
│ ├── jetson-agx-xavier-devkit-R32x-R35i.conf -> p2822-0000+p2888-0004-R32x-R35i.conf
│ ├── jetson-agx-xavier-devkit-R35A-R35i.conf -> p2822-0000+p2888-0004-R35A-R35i.conf
│ ├── jetson-xnx-devkit-emmc-R32i.conf -> p3509-0000+p3668-0001-qspi-emmc-R32i.conf
│ ├── jetson-xnx-devkit-emmc-R32x-R35i.conf -> p3509-0000+p3668-0001-qspi-emmc-R32x-R35i.conf
│ ├── jetson-xnx-devkit-emmc-R35A-R35i.conf -> p3509-0000+p3668-0001-qspi-emmc-R35A-R35i.conf
│ ├── p2822-0000+p2888-0004-R32x-R35i.conf
│ ├── p2822-0000+p2888-0004-R35A-R35i.conf
│ ├── p2822-0000+p2888-0008-R32x-R35i.conf
│ ├── p2822-0000+p2888-0008-R35A-R35i.conf
│ ├── p3509-0000+p3668-0001-qspi-emmc-R32i.conf
│ ├── p3509-0000+p3668-0001-qspi-emmc-R32x-R35i.conf
│ ├── p3509-0000+p3668-0001-qspi-emmc-R35A-R35i.conf
│ ├── Tegra_Software_License_Agreement-Tegra-Linux.txt
│ └── tools
│ └── ota_tools
│ └── version_upgrade
│ ├── BOOTAA64_r35_1.efi
│ ├── build_base_recovery_image.sh
│ ├── flash_l4t_t194_nvme_for_r32.xml
│ ├── flash_l4t_t194_nvme_rootfs_ab_for_r32.xml
│ ├── Image_based_OTA_Examples.txt
│ ├── kernel_bootctrl.bin.normal
│ ├── kernel_bootctrl.bin.reset
│ ├── kernel_bootctrl.bin.update
│ ├── l4t_generate_ota_package.sh
│ ├── l4t_gen_uefi_sb_overlay.sh
│ ├── l4t_update_partitions.sh
│ ├── NOTICE.uefi
│ ├── nv_ota_check_version.sh
│ ├── nv_ota_common.func
│ ├── nv_ota_common_utils.func
│ ├── nv_ota_customer.conf
│ ├── nv_ota_decompress_package.sh
│ ├── nv_ota_log.sh
│ ├── nv_ota_preserve_data.sh
│ ├── nv_ota_rootfs_updater.sh
│ ├── nv_ota_run_tasks.sh
│ ├── nv_ota_start.sh
│ ├── nv_ota_update_all_in_recovery.sh
│ ├── nv_ota_update_alt_part.func
│ ├── nv_ota_update_rootfs_in_recovery.sh
│ ├── nv_ota_update.sh
│ ├── nv_ota_update_with_layout_change.sh
│ ├── nv_ota_update_without_layout_change.sh
│ ├── nv_ota_validate.sh
│ ├── nv_update_alt_part.sh
│ ├── ota_backup_files_list.txt
│ ├── ota_board_specs.conf
│ ├── ota_check_partitions.sh
│ ├── ota_multi_board_specs.sh
│ ├── ota_validate_params.sh
│ ├── uefi_jetson_r35_1.bin
│ ├── upgradetasklist.txt.jetson-agx-xavier-devkit.R32x_to_R35-ToT_emmc
│ ├── upgradetasklist.txt.jetson-agx-xavier-industrial.R32x_to_R35-ToT_emmc
│ └── upgradetasklist.txt.jetson-xavier-nx-devkit-emmc.R32x_to_R35-ToT_emmc
└── ota_tools_R35.6.0_aarch64.tbz2
- Applied the OTA bootloader only payload to the NVME:
root@JETSONE1:/boot/ota/ota-tools/Linux_for_Tegra/tools/ota_tools/version_upgrade# sudo ./nv_ota_start.sh /boot/ota/ota_payload_package.tar.gz
Command: ./nv_ota_start.sh /boot/ota/ota_payload_package.tar.gz
Current rootfs is on /dev/nvme0n1
init_ota_log /ota_log
Creating log dir at /ota_log
Create log file at /ota_log/ota_20250709-111906.log
OTA_LOG_FILE=/ota_log/ota_20250709-111906.log
Extract /boot/ota/ota_payload_package.tar.gz
update_nv_boot_control_in_rootfs /ota_work
3668-301---1--jetson-xavier-nx-devkit-emmc-
Info. Installing mtdblock.
Info. Active boot storage: nvme0n1
Info. Legacy mode: false
TNSPEC 3668-301-0001-E.0-1-2-jetson-xavier-nx-devkit-emmc-
COMPATIBLE_SPEC 3668-301---1--jetson-xavier-nx-devkit-emmc-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE nvme0n1
TEGRA_EMMC_ONLY false
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0
Info: Write TegraPlatformCompatSpec with 3668-301---1--jetson-xavier-nx-devkit-emmc-.
Info. Uninstalling mtdblock.
check_prerequisites
decompress_ota_package ota_package.tar /ota_work
decompress_ota_package: start at Wed Jul 9 11:19:11 UTC 2025
Sha1 checksum for /ota_work/ota_package.tar (2efbc7cd0056c0c9e03dec6386b78380ef34a153) matches
decompress_ota_package: end at Wed Jul 9 11:19:14 UTC 2025
nv_ota_update_without_layout_change.sh
Command: nv_ota_update_without_layout_change.sh
check_target_board /ota_work TARGET_BOARD
get_chip_id CHIP_ID
ota_choose_images /ota_work
COMPATIBLE_SPEC=3668-301---1--jetson-xavier-nx-devkit-emmc-
TEGRA_CHIPID=0x19
_BOARD_SPEC_NAME=3668-301--
Copy files from ./images-R35-ToT/3668-301--/ to ./images-R35-ToT/
is_rootfs_a_b_enabled ROOTFS_AB_ENABLED ROOTFS_CURRENT_SLOT
ROOTFS_AB_ENABLED=0
ROOTFS_CURRENT_SLOT=0
is_rootfs_encryption_enabled ROOTFS_ENC_ENABLED
ROOTFS_ENC_ENABLED=1
get_update_slot UPDATE_SLOT
UPDATE_SLOT=B
get_update_control /ota_work UPDATE_BOOTLOADER UPDATE_ROOTFS
UPDATE_BOOTLOADER=1, UPDATE_ROOTFS=0
select_uefi_capsule /ota_work
check_bootloader_version /ota_work
update_bootloader /ota_work
trigger_uefi_capsule_update /ota_work jetson-xavier-nx-devkit-emmc
Mount esp partition on /opt/nvidia/esp
Copying /ota_work/TEGRA_BL.Cap into /opt/nvidia/esp/EFI/UpdateCapsule
Triggering UEFI capsule update by writing \x07\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00 to UEFI variable OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c
dd if=/tmp/var_tmp.bin of=OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c bs=12
1+0 records in
1+0 records out
12 bytes copied, 0.032465 s, 0.4 kB/s
Bootloader on non-current slot(B) is to be updated once device is rebooted
clean_up_ota_files
-
sudo reboot - Reboot the jetson
-
When jetson is rebooted, an update progress bar screen pops up for the OTA BOOTLOADER update, but when it is 100% done, the jetson goes to Recovery mode 7e19. (See attached picture)
Is the process correct ? or I am missing something ?
Before reboot, jetson status after applying payload :
cat /ota_work/update_control
bootloader
nvbootctrl dump-slots-info
Current version: 35.4.1
Capsule update status: 0
Current bootloader slot: A
Active bootloader slot: A
num_slots: 2
slot: 0, status: normal
slot: 1, status: normal
root@JETSONE1:/boot/ota/ota-tools/Linux_for_Tegra/tools/ota_tools/version_upgrade# cat /etc/nv_tegra_release
# R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug 1 19:57:35 UTC 2023
I have been trying to update the bootloader using Bootloader Capsule Payload this link:
- I generated the TEGRA_BL.cap with PKC and SBK keys as below:
$ sudo ./l4t_generate_soc_bup.sh -u ../keys/jetson.pem -v ../keys/sbk_key.txt -b jetson-xavier-nx-devkit-emmc t19x
$ ./generate_capsule/l4t_generate_soc_capsule.sh -i bootloader/payloads_t19x/bl_only_payload -o ./TEGRA_BL.Cap t194
- Manually trigger the BL Payload on jetson ; this time I booted from NVME as active boot media . After reboot, the Capsule is triggered and Bootloader could be updated with “Update progress” bar.
$ sudo mkdir -p /opt/nvidia/esp
$ esp_uuid=$(lsblk -o name,partlabel,uuid | grep "nvme0n1" | awk '{ if($2 == "esp") print $3 }')
$ sudo mount UUID=$esp_uuid /opt/nvidia/esp
sudo mkdir -p /opt/nvidia/esp/EFI/UpdateCapsule
sudo cp TEGRA_BL.Cap /opt/nvidia/esp/EFI/UpdateCapsule
$ cd /sys/firmware/efi/efivars/
$ printf "\x07\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00" > /tmp/var_tmp.bin
$ sudo dd if=/tmp/var_tmp.bin of=OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c bs=12;sync
$ sudo reboot
- I did the same steps in step 2; but now I booted up from bootable USB. The idea is to update the esp partition of the NVME with BLCapsule while being booted up from USB along with rootfs on NVME. Current active boot media is USB; done the below steps, rebooted the jetson and then try to boot from NVME on whose esp partition the BL Capsule has been copied . Why is this the case ? How does the Bootloader capsule update got triggered?
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 16M 1 loop
sda 8:0 1 119.3G 0 disk
├─sda1 8:1 1 19.8G 0 part /
├─sda2 8:2 1 128M 0 part
├─sda3 8:3 1 768K 0 part
├─sda4 8:4 1 31.6M 0 part
├─sda5 8:5 1 128M 0 part
├─sda6 8:6 1 768K 0 part
├─sda7 8:7 1 31.6M 0 part
├─sda8 8:8 1 80M 0 part
├─sda9 8:9 1 512K 0 part
├─sda10 8:10 1 100M 0 part
├─sda11 8:11 1 64M 0 part
├─sda12 8:12 1 80M 0 part
├─sda13 8:13 1 512K 0 part
├─sda14 8:14 1 64M 0 part
└─sda15 8:15 1 400M 0 part
mmcblk0 179:0 0 14.7G 0 disk
├─mmcblk0p1 179:1 0 14G 0 part
├─mmcblk0p2 179:2 0 64M 0 part
├─mmcblk0p3 179:3 0 448K 0 part
├─mmcblk0p4 179:4 0 32.6M 0 part
├─mmcblk0p5 179:5 0 2.5M 0 part
├─mmcblk0p6 179:6 0 64K 0 part
├─mmcblk0p7 179:7 0 1M 0 part
├─mmcblk0p8 179:8 0 1M 0 part
├─mmcblk0p9 179:9 0 1M 0 part
├─mmcblk0p10 179:10 0 1.5M 0 part
├─mmcblk0p11 179:11 0 1M 0 part
├─mmcblk0p12 179:12 0 64M 0 part
├─mmcblk0p13 179:13 0 448K 0 part
├─mmcblk0p14 179:14 0 32.5M 0 part
├─mmcblk0p15 179:15 0 80M 0 part
├─mmcblk0p16 179:16 0 512K 0 part
├─mmcblk0p17 179:17 0 100M 0 part
├─mmcblk0p18 179:18 0 64M 0 part
├─mmcblk0p19 179:19 0 80M 0 part
├─mmcblk0p20 179:20 0 512K 0 part
├─mmcblk0p21 179:21 0 64M 0 part
└─mmcblk0p22 179:22 0 101M 0 part
zram0 251:0 0 569.4M 0 disk [SWAP]
zram1 251:1 0 569.4M 0 disk [SWAP]
zram2 251:2 0 569.4M 0 disk [SWAP]
zram3 251:3 0 569.4M 0 disk [SWAP]
zram4 251:4 0 569.4M 0 disk [SWAP]
zram5 251:5 0 569.4M 0 disk [SWAP]
nvme0n1 259:0 0 119.2G 0 disk
├─nvme0n1p1 259:1 0 400M 0 part
├─nvme0n1p2 259:2 0 39.6G 0 part
├─nvme0n1p3 259:3 0 128M 0 part
├─nvme0n1p4 259:4 0 768K 0 part
├─nvme0n1p5 259:5 0 31.6M 0 part
├─nvme0n1p6 259:6 0 128M 0 part
├─nvme0n1p7 259:7 0 768K 0 part
├─nvme0n1p8 259:8 0 31.6M 0 part
├─nvme0n1p9 259:9 0 80M 0 part
├─nvme0n1p10 259:10 0 512K 0 part
├─nvme0n1p11 259:11 0 100M 0 part
├─nvme0n1p12 259:12 0 64M 0 part
├─nvme0n1p13 259:13 0 80M 0 part
├─nvme0n1p14 259:14 0 512K 0 part
├─nvme0n1p15 259:15 0 64M 0 part
└─nvme0n1p16 259:16 0 400M 0 part
$ sudo mkdir -p /opt/nvidia/esp
$ esp_uuid=$(lsblk -o name,partlabel,uuid | grep "nvme0n1" | awk '{ if($2 == "esp") print $3 }')
$ sudo mount UUID=$esp_uuid /opt/nvidia/esp
sudo mkdir -p /opt/nvidia/esp/EFI/UpdateCapsule
sudo cp TEGRA_BL.Cap /opt/nvidia/esp/EFI/UpdateCapsule
$ cd /sys/firmware/efi/efivars/
$ printf "\x07\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00" > /tmp/var_tmp.bin
$ sudo dd if=/tmp/var_tmp.bin of=OsIndications-8be4df61-93ca-11d2-aa0d-00e098032b8c bs=12;sync
#Additional update of NVME rootfs via USB
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml -S 40GiB --direct nvme0n1 --showlogs jetson-xavier-nx-devkit-emmc external
$ sudo reboot
# Disconnected the USB stick and Next boot is from NVME itself. But the BLCapsule update didn't get trigger. Why ?
I could see one error in nvbootctrl :
sudo nvbootctrl dump-slots-info
Error: config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Current version: 35.6.0
Capsule update status: 0
Current bootloader slot: B
Active bootloader slot: B
num_slots: 2
slot: 0, status: normal
slot: 1, status: normal
cat /etc/nv_boot_control.conf
TNSPEC ----1-2-jetson-xavier-nx-devkit-emmc-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0
