Boot fails after restoring a Jetson orin NX

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
,
1

Hello Team,
I have a Nvidia Orin NX (Not using Nvidia’s carrier board) that has Root AB and Disk Encryption enabled. Jetpack version is 6.2. After backing up, I tried to restore another same spec device. Eventhough the restoration process was showing successful the device doesn’t boot. A picture is attached which shows where its stuck during boot.

My question is, when a device has encrypted disk partitions is there any different approach of using backup_restore process?

would be helpful if you could share some command that shows the correct approach.

thanks.

IMG_3606
IMG_36061920×1194 294 KB

3

hello tanzelur,

did you Creating Encrypted Images with a Generic Key?
otherwise, it’s by default to unlock the encrypted root device with the per-device unique passphrase (i.e. ECID).

4

Hi JerryChang,
I have generated eks_t234.img and sym2_t234.key to encrypt the disk. No generic passphrase was defined to the device.
thanks

5

hello tanzelur,

please enable generic passphrase if you would like to restore the golden image to other devices. (even though it’s the same spec)

6

Hi @JerryChang,
As per your suggestion, I flashed a Jetson Orin NX device with generic passphrase and then backed up for my golden image by using backup_restore method. The commands I used:

sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" --no-flash --network usb0 p3509-a02-p3767-0000 internal

sudo ROOTFS_AB=1 ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./sym2_t234.key ``-p "--generic-passphrase"`` -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_ab_enc.xml --external-only --append --network usb0 p3509-a02-p3767-0000 external

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only

# Backup Generation
sudo ./tools/backup_restore/l4t_backup_restore.sh -e nvme0n1 -b p3509-a02-p3767-0000

Then I tried to create an image from the backup image and flash another device again. Unfortunately this method did not work. commands for the reflash:

sudo ROOTFS_ENC=1 ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --use-backup-image --no-flash --network usb0 -i ./sym2_t234.key ''-p "--generic-passphrase"'' p3509-a02-p3767-0000 nvme0n1

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only

May be I give you an overview about what I want to achieve.
My goal is to develop an image with all my installations that needs to run my product’s software and use the same image in production. The most important Criteria is, the disk has to be encrypted.

And the reason I want to keep the same disk encryption key is, in order to make an OTA update, we need to generate a payload. while this payload package generation it needs the disk encryption key.

If I use different encryption keys for each device, its not practical to produce device specific payload for OTA. It should be one payload that can be be pushed to all devices in the field.

That is why I am trying to use backup and restore method.

Kindly guide me about this issue. What are the things I am doing wrong in order to achieve this?
Is this even possible to re-flash multiple device from a backed up Jetson?

Thanks in advance.

8

Hi tanzelur,

Please try do below step first before your backup your golden image and check issue whether exists.

Enter this command on the device
$ sudo sed -i "/\/boot\/efi/d" /etc/fstab

Thanks

9

hello @DavidDDD,
I’ve executed this sudo sed -i "/\/boot\/efi/d" /etc/fstab step before backing up my device.
After that I executed the commands below,

sudo ./tools/backup_restore/l4t_backup_restore.sh -e nvme0n1 -b -c  p3509-a02-p3767-0000

sudo ROOTFS_ENC=1 ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --use-backup-image --no-flash --network usb0 -i ./sym2_t234.key ''-p "--generic-passphrase"'' --massflash 2  p3509-a02-p3767-0000 nvme0n1

cd mfi_p3509-a02-p3767-0000/
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 2

However, the issue still persists. Device booting stuck with **ERROR: fail to unlock the encrypted dev /dev/nvme0n1p3**this Error.

The flash log indicates that the flashing process was successful, but from the boot log you can see that the problem remains unresolved.

I’m attaching both the flash log and the boot log for your reference.

Please let me know if there are any other approaches I can try.

Thanks.
boot-log.log (34.7 KB)
flash_3-7_1_20250519-135719.log (151.8 KB)

10

hello tanzelur,

let’s try to flash only QSPI for updating your EKS image, please also have new generated eks_t234.img applied.
for instance,
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh -i ./sym2_t234.key "-c bootloader/generic/cfg/flash_t234_qspi.xml --no-systemimg" jetson-orin-nano-devkit internal

11

Hi,
unfortunately, I could not use my backup images to massflash my other Jetsons. :(

So, at the moment, I took a different approach of using customized ROOTFS (by copying a rootfs from a working Jetson). This way, both of my test devices are successfully booting and reflecting my customization’s at both Root A/B partition.

However, I ended up having an issue of OTA update which is my another goal to achieve.

While generating the massflash image, I used generic-passphrase.
Here is this command:

sudo ROOTFS_AB=1 ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./sym2_t234.key ``-p "--generic-passphrase"``  --massflash 2 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_ab_enc.xml --external-only --append --network usb0 p3509-a02-p3767-0000 external

and the same way, I have included the disk encryption key while generating OTA payload package,

sudo -E ROOTFS_ENC=1 ROOTFS_AB=1 ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -s -r -i ./sym2_t234.key --external-device nvme0n1 p3509-a02-p3767-0000 R36-4

The ota payload generation was successful, but when I trigger the ota update process it got failed.

Encrypted partition is unlocked at /dev/mapper/crypt_root_other
Failed to run "umount /mnt/crypt_root_other"
Failed to run "get_unlocked_devnode /dev/nvme0n1p3 rootfs_part_devnode"
Failed to run "update_rootfs_runtime /ota_work A external_device"
Failed to run "update_rootfs_with_a_b_enabled /ota_work"
Failed to run "update_rootfs /ota_work"

here is the full log:
ota_20250520-112330.log (7.7 KB)

So, wanted to know if i use generic-passphrase, is it possible to create a common ota payload that includes the disk encryption key to update my device?
Expecting your kind help.

Thanks in advance.

15

hello tanzelur,

let me have confirmation.
you can perform backup/restore (including ROOTFS_AB=1 ROOTFS_ENC=1) on the same platform,
however, you’ve booting up failure when you try to backup from Orin-NX-A and restore to Orin-NX-B, (crossing platforms) right?

17

Hello JerryChang,

Yes, that’s correct.
Every time it works on the same device from which I backed up. (Orin-NX-A).

however, Orin-NX-B can Not boot up, and shows the error in the picture shared before.
(Tried methods using - r option with l4t_backup_restore.sh. And also with l4t_initrd_flash.sh –use-backup-image)

Thanks

18

Hello @JerryChang,
Could you have a look at the OTA trigger issue, please.
Why usually these failures occur?

I have checked now, even for a single Jetson ORIN NX device that flashed with disk encryption key ( without --generic-passphrase included ) is throwing the same Failure message.

ota_20250521-163036.log (7.5 KB)

Command used to prepare the ota payload is, ( used this<Linux_for_Tegra>/rootfs to update the rootfs, so no -s).

The generated ota_payload worked before perfectly, and I had no issue with the ota update.

now its throwing those failures. :(

Please, suggest how to solve this. its really really important at the moment.

Thanks. Looking forward to your answer.

19

hello tanzelur,

I would like to have this thread for tracking backup/restore crossing platforms.
for instance, backup from Orin-NX-A and restore to Orin-NX-B, which has ROOTFS_AB=1 ROOTFS_ENC=1 included.

it’s my suggestion to have another new forum thread for OTA specific issue.

21

hello tanzelur,

let’s keep this discussion thread for backup/restore with disk encryption enabled.

please use -e mmcblk0:nvme0n1 to back up both internal and external storage from Orin-NX-A.

22

hi @JerryChang ,
thanks for your response, I’ll try this and let you know the updates. :)

23

Is this still an issue to support? Any result can be shared?

24

Hi,
Unfortunately that didn’t work either. I am sharing a snap of the flash_log, how the backup_restore ended up.

***************************************
*                                     *
*  Step 3: Start the flashing process *
*                                     *
***************************************
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for device to expose ssh ......Waiting for device to expose ssh ...Device has booted into initrd. You can ssh to the target by the command:
$ ssh root@fc00:1:1:0::2
Cleaning up...
Log is saved to Linux_for_Tegra/initrdlog/flash_3-7_0_20250604-174432.log 
Run command: 
ln -s /proc/self/fd /dev/fd && mount -o nolock [fc00:1:1::1]:/home/tanzelur/TbA_WS/JetsonLinux/JetsonMVP-prod/Linux_for_Tegra/tools/backup_restore /mnt && /mnt/nvbackup_partitions.sh -e mmcblk0:nvme0n1 -n  && echo Backup image is stored in /home/tanzelur/TbA_WS/JetsonLinux/JetsonMVP-prod/Linux_for_Tegra/tools/backup_restore/images
 on root@fc00:1:1::2
fdisk: cannot open /dev/mmcblk0: No such file or directory

Command to initiate the process:

sudo ./tools/backup_restore/l4t_backup_restore.sh -e mmcblk0:nvme0n1 -b -c p3509-a02-p3767-0000

Thanks again for replying :)

25

hello tanzelur,

umm.. backup/restore may not works with disk encryption enabled,
according to developer guide, Creating Encrypted Images with a Generic Key. it’s massflash image created from host machine instead from the target.

let me arrange resources for checking internally.

26

hello tanzelur,

it’s backup/restore script limitation with ROOTFS_ENC devices.
it must to run flash script (from host machine) to deploy devices with disk encryption for using a generic key.
here’re brief steps.
a) add --massflash option to generate a massflash package,

$ sudo BOARDID=3767 BOARDSKU=0000 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --no-flash jetson-orin-nano-devkit internal
$ sudo BOARDID=3767 BOARDSKU=0000 ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 --showlogs  --no-flash --external-device nvme0n1p1 -S 16GiB -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --append -i ./disk_enc.key ``-p "--generic-passphrase"`` --massflash 2 jetson-orin-nano-devkit external

b) it’s a massflash (mfi_.tar.gz) package, by extracting this package, you should then putting two Orin Nano DevKits into recovery mode, and flashing them simultaneously to enable disk encryption.

$ sudo tar xpfv mfi_<target-board>.tar.gz. 
$ cd mfi_<target-board> 
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 2
28

hi JerryChang,
Sorry again for replying very late. I was busy with other things meanwhile.

massflash method works. I have tested with two Jetson. Priority was to make a clone through backup_restore, however massflash is also fine at the moment.

thank you.