Disk Encryption on Orin NX NVMe Not working

hello _Becu,

you meant you’re using this test key? $ echo "f0e0d0c0b0a001020304050607080900" > sym2_t234.key?

you don’t need to update eks image because that’s same as default image.
please try following below, to confirm you’re given keys correctly, and running correct xml configuration file for image flashing.
for example,
$ echo "f0e0d0c0b0a001020304050607080900" > sym2_t234.key
$ sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -i ./sym2_t234.key --no-flash --showlogs --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 100GiB --append --external-only jetson-orin-nano-devkit external

Hello,

Yes, again, I didn’t do any modifications to the example.sh. I’m going through the entire flow, as if I’m using custom keys, to validate that the steps work.

Linux_for_Tegra$ cat sym2_t234.key 
f0e0d0c0b0a001020304050607080900

OPTEE passes the key extraction process, so I’m assuming the key and EKB are correct.

how about re-flash the target with above commands to check again?

Should I run your command first, then flash with the one that worked for me in my previous comment?
So

  1. sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -i ./sym2_t234.key --no-flash --showlogs --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 100GiB --append --external-only jetson-orin-nano-devkit external
  2. sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --external-device nvme0n1p1 -i ./sym2_t234.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --network usb0 jetson-orin-nano-devkit external

I see the one you provided has --no-flash in it

you may running with --no-flash to create image blob locally, and enabling --flash-only to flash the target with two step approach.
or, please exclude --no-flash from the command-line to flash the target directly.

The command is similar to the one I used in Attempt 4, but has --append, which is Only applicable when using with --no-flash --external-only option.. I’m not using --no-flash, so I’ll also remove the --append. The only difference remains the extra -S 100GiB then.

In Attempt 4 I managed to succesfully flash the device and boot past the key extraction process, but it failed with mount: /mnt: special device /dev/mmcblk0p1 does not exist . Do you think the issue is with the partition layout?

Ran your command, but it kept failing during the flash with

[   4.8709 ] End sector for APP_ENC, expected at: 122159070, actual: 0
[   4.8709 ] 
Error: Return value 4

as I did not modify the flash_l4t_t234_nvme_rootfs_enc.xml to accomodate the 100GiB partition.

Modified your command from 100GiB to 8GiB:
sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -i ./sym2_t234.key --showlogs --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 8GiB --external-only jetson-orin-nano-devkit external

Steps:

  1. rm -rf Linux_for_Tegra
  2. re-create Linux_for_Tegra from sources, apply binaries, create default user
  3. cd Linux_for_Tegra
  4. copy eks img and sym2_t234.key
  5. sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -i ./sym2_t234.key --showlogs --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 8GiB --external-only jetson-orin-nano-devkit external

The result is the same as in Attempt 4. Device gets flashed, boots and then fails with the same error:

EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
OTA work directory /mnt/ota_work is not found on /dev/nvme0n1p1
lock_encrypted_partition dm_crypt_ota
Finding OTA work dir on internal storage device
mount /dev/mmcblk0p1 /mnt
mount: /mnt: special device /dev/mmcblk0p1 does not exist.
Failed to mount /dev/mmcblk0p1 on the /mnt
Failed to run "mount_ota_work_partition /dev/mmcblk0p1 /mnt"
OTA work directory is not found on internal and external storage devices

Hi, could you give some proper instructions about Attempt 3, please?

HI,

Check your UART logs, the setup that worked for me had the MB1, OPTEE and UEFI versions from March 2023. These are the August versions, for reference:
MB1 (version: 1.2.0.0-t234-54845784-562369e5)
OP-TEE version: 3.21 (gcc version 9.3.0 (Buildroot 2020.08)) #2 Tue Aug 1 19:39:55 UTC 2023 aarch64
Jetson UEFI firmware (version 4.1-33958178 built on 2023-08-01T19:34:02+00:00)

Follwing the steps in UEFI Secureboot updated the March versions to the August ones.

Other than that, steps are the same as the ones I referenced above in “Steps” but with the flash command from Attempt 2.

IMO now that the device boots, these versions are not the issue.

So, you managed to enable disk encryption with this? I mean does your work lead to the solution of your problem(and encryption at all)?

Yes, device booted and disk encryption was working, but it isn’t officially supported on the Orin NX (from what the docs say). Also Secureboot/ UEFI Secureboot are also not officially supported on the NX with those versions.

On second thought, the issue with the system not booting properly from Attempt 4 might be related to the versions, as there’s a log with L4TLauncher: Attempting Recovery Boot.

Ok, thanks for your work!

@JerryChang Do you have any ideas on why it might attempt to boot into recovery right on the first boot after flashing? Or any others things to try?

Hi @_Becu in your previous post, you mentioned “The result is the same as in Attempt 4. Device gets flashed, boots and then fails with the same error:”, did you get that resolved? a bit confusing when you mentioned “device booted and disk encryption was working”
Thanks

hello _Becu,

we’ve test again locally to confirm disk encryption is working.

here’re our test steps for your reference,
Preparation: $ sudo apt-get install cryptsetup.
note, it’s cryptsetup utility to create encrypted rootfs for image flashing.

  1. Generate images for QSPI:
    $ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --no-flash --network usb0 jetson-orin-nano-devkit internal
  2. Generate the key:
    $ echo "f0e0d0c0b0a001020304050607080900" > ekb.key
  3. Generate images for external storage device:
    $ sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./ekb.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --append --network usb0 jetson-orin-nano-devkit external
  4. Flash images into the both storage devices:
    $ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only

Flash success and boot up:

$ df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/crypt_root   54G  5.6G   46G  12% /
/dev/mapper/crypt_UDA   374M   14K  350M   1% /mnt/crypt_UDA
none                    7.5G     0  7.5G   0% /dev
tmpfs                   7.6G   36K  7.6G   1% /dev/shm
tmpfs                   1.6G   19M  1.5G   2% /run
tmpfs                   5.0M  4.0K  5.0M   1% /run/lock
tmpfs                   7.6G     0  7.6G   0% /sys/fs/cgroup
/dev/nvme0n1p1          371M   97M  247M  29% /boot
tmpfs                   1.6G   72K  1.6G   1% /run/user/1000

Hi @JerryChang. this method works if I’m using “f0e0d0c0b0a001020304050607080900” key. Another value(random) gives ERROR: fail to unlock the encrypted dev /dev/nvme0n1p2. in UART logs. Is it possible to use another key value to make it work?

P.S. Key was generated with sudo openssl rand -rand /dev/urandom -hex 16 > ekb.key

Thanks for the confirmation, @JerryChang. I did try locally with mixed results. Some attempts worked, so that’s a win.

Setup: Orin Nano carrier board, 2x Orin NX SoMs which I didn’t experiment with in the disk encryption setup, they were previously flashed without any security features with JP 5.1.2.

  1. Clean Linux_for_Tegra. Only your commands from above.
  • Orin SoM #1
    • flashed and booted succesfully with encryption working.
  • Switched the SoM to Orin #2.
    • Ran only command 4.
    • Orin booted but hung with ERROR: fail to unlock the encrypted dev /dev/nvme0n1p2.

  1. Clean Linux_for_Tegra. Again only your commands.
  • Orin SoM #2
    • worked, booted succesfully.
    • In the same environment, run all the commands again
    • Command 4 failed with Error: Could not stat device /dev/mmcblk0 - No such file or directory.. This replicates every time you run the commands again.

  1. Clean Linux_for_Tegra. Again only your commands.
  • Switched back to Orin SoM #1
    • Booted, but in recovery mode from the 1st boot L4TLauncher: Attempting Recovery Boot
    • Failed later with
[    7.494074] Root device found: initrd
modprobe: FATAL: Module r8168 not found in directory /lib/modules/5.10.120-tegra
[    7.496146] Mount initrd as rootfs and enter recovery mode
Finding OTA work dir on external storage devices
Checking whether device /dev/mmcblk?p1 exist
Device /dev/mmcblk?p1 does not exist
Checking whether device /dev/sd?1 exist
Device /dev/sd?1 does not exist
Checking whether device /dev/nvme?n1p1 exist
Looking for OTA work directory on the device(s): /dev/nvme0n1p1
mount /dev/nvme0n1p1 /mnt
[    7.520214] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null)
is_boot_only_partition /mnt
The mounted /dev/nvme0n1p1 is boot partition, try locating rootfs partition and mount it...
mount_rootfs_partition /dev/nvme0n1p1 /mnt
Found encrypted rootfs partition /dev/nvme0n1p2 through UUID(aa83d09a-41cb-4239-bebd-f682d7814032)
umount /mnt
unlock_encrypted_partition /dev/nvme0n1p2 dm_crypt_ota dm_crypt
is_luks_partition /dev/nvme0n1p2
is_unlocked /dev/nvme0n1p2 unlocked_device_name
get_uuid_for_luks_partition /dev/nvme0n1p2 luks_uuid
No key available with this passphrase.
Failed to unlock the LUKS partition /dev/nvme0n1p2(UUID=aa83d09a-41cb-4239-bebd-f682d7814032)
Failed to run "unlock_encrypted_partition /dev/nvme0n1p2 dm_crypt_ota dm_crypt"
Failed to run "moutn_rootfs_partition /dev/nvme0n1p1 /mnt"
Failed to run "mount_ota_work_partition /dev/nvme0n1p1 /mnt"
Finding OTA work dir on internal storage device
mount /dev/mmcblk0p1 /mnt
mount: /mnt: special device /dev/mmcblk0p1 does not exist.
Failed to mount /dev/mmcblk0p1 on the /mnt
Failed to run "mount_ota_work_partition /dev/mmcblk0p1 /mnt"
OTA work directory is not found on internal and external storage devices

  1. Clean Linux_for_Tegra. Again only your commands.
  • Orin SoM #1
    • Command 4 failed with Error: Could not stat device /dev/mmcblk0 - No such file or directory.
  • Switch to SoM #2
    • Run only command #4. Same error.

So it seems that the commands work…sometimes :-?. But sometimes fail even with a clean environment.
I’ll try to change to a custom key and add Secureboot to the setup, see how that goes.

Thanks for your help!

Hi @bingnvidia, Attempt 3 which I was referring to in that comment involved mixing the bootloader versions from a previous jetpack release with the external storage images built on JP 5.1.2. So a hacky solution, that’s why I mentioned it’s only an “FYI”.

@alex.iakov1337 See Disk Encryption — Jetson Linux Developer Guide documentation for custom keys. You have to create and update eks.img as well.

Thank you very much! Didn’t notice)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.