Disk Encryption on Orin NX NVMe Not working

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
, ,
1

Hello,

I’m using an Orin NX 16GB on a Orin Nano carrier board with no fuses burnt. The goal is to enable Disk Encryption, as it’s supported with JP 5.1.2. Went through existing threads, Docs, script readme’s, etc. Nothing seems to work on the latest releases.

Attempts: (clean environments every time). Listing only the most relevant ones.

Attempt 1

Steps mentioned in Orin NX disk encryption - #17 by carolyuu
Result: Flash doesn’t work.

Error: Could not stat device /dev/mmcblk0 - No such file or directory., as mentioned in the reply in that post as well.

Attempt 2

Workflow 10 from README_initrd_flash.txt

Keys generated with source/public/nvidia-jetson-optee-source/optee/samples/hwkey-agent/host/tool/gen_ekb/example.sh. Copied eks_t234.img to bootloader/eks_t234.img and sym2_t234.key to Linux_for_Tegra
sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml -p "-i ./sym2_t234.key" --external-only -S 8GiB jetson-orin-nano-devkit external

Result: Flash works, Device starts booting, but fails. UART logs:
nofuses_defaultkeys.log (39.8 KB)

E/TC:00 00 ekb_extraction_process:319 Tried all EKB_RKs but still can't extract the EKB image.

Attempt 3

More as an FYI, the only setup that worked was before I attempted UEFI Secureboot - which updated MB1, OPTEE and UEFI from their March '23 version to the August '23 ones. In this setup, with the commands identical to the ones above, Disk Encryption seemed to work, and the target succesfully booted up. This setup is not officially supported, and also does not (officially) support Secure Boot on Orin NX.

What is the proper way to achieve a working setup with Disk Encryption enabled?

@carolyuu @JerryChang I saw you had some useful insights into related issues, if you could help us on this.

Thanks!

3

hello _Becu,

did you replace any keys within example.sh to generate a new eks_t234.img? may I know what’s the key you’ve updated.

furthermore,
you should also try partition flash only to update EKS partition, A_eks.
for example,
(1) Running with flash script to create eks_t234_sigheader.img.encrypt.
$ sudo ./flash.sh --no-flash -r -k A_eks jetson-agx-orin-devkit mmcblk0p1
(2) Copy sign/encrypt binary to… $OUT/Linux_for_Tegra/tools/kernel_flash/images/internal/
(3) Execute below command to update EKS partition for your Orin NX.
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh -k A_eks --flash-only --showlogs

4

Hello @JerryChang,

Thank you for your reply!

I did not change any values in example.sh, everything was done in a freshly downloaded environment to avoid carrying over errors from previous attempts.

I tried running the commands above, but:

(1) failed with Reusing existing system.img... file does not exist.
Tried skipping the -r option, seemed to work. Also replaced jetson-agx-orin-devkit with jetson-orin-nano-devkit, logs here (8.8 KB)

(2) Done
(3) Seemed to work, logs here (10.3 KB)

Booting however failed after the same error: ekb_extraction_process:319 Tried all EKB_RKs but still can't extract the EKB image.

To sanity check these steps, I did edit eks_t234.img before running (1) (2) and (3) again, and I did see a change in the size mentioned by (3) here: Added binary blob_eks_t234_sigheader.img.encrypt of size 9232 from 9232 to 9434, so the steps seem to work.

5

Looked through the OP-TEE (version 3.21) source code and we observed that the variable ekb_rks_num is set to 0 at the beginning and not changed. The for loop which is supposed to derive the other keys and decrypt the EKB never actually runs.

LE: missed the ekb_rks_num++. Looking if the EKB_RK is correct, although not sure this is the right path to proceed

Linux_for_Tegra/source/public/nvidia-jetson-optee-source$ grep -rnw './' -e 'ekb_rks_num'
./optee/optee_os/core/pta/tegra/jetson_user_key_pta.c:59:static uint8_t ekb_rks_num = 0;
./optee/optee_os/core/pta/tegra/jetson_user_key_pta.c:215:	for (i = 0; i < ekb_rks_num; i++) {
./optee/optee_os/core/pta/tegra/jetson_user_key_pta.c:318:	if (i == ekb_rks_num) {
./optee/optee_os/core/pta/tegra/jetson_user_key_pta.c:346:		rc = tegra_se_aes_ecb_kdf(ekb_rks[ekb_rks_num], sizeof(ekb_rks[ekb_rks_num]),
./optee/optee_os/core/pta/tegra/jetson_user_key_pta.c:351:			ekb_rks_num++;
./optee/optee_os/core/pta/tegra/jetson_user_key_pta.c:358:	if (ekb_rks_num == 0) {
6

One step closer:

Attempt 4

(probably 104)

  1. Clean Linux_for_Tegra environment (multiple builds/flashes seem to affect subsequent attemps, guess some files don’t rebuild every time).
  2. Copy eks_t234.img and sym2_t234.key from the example.sh folder to bootloader/ and Linux_for_Tegra/ respectively
  3. sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --external-device nvme0n1p1 -i ./sym2_t234.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --network usb0 jetson-orin-nano-devkit external

Flash succeeds, Orin boots, but hangs at some point:

[    7.484606] Run /init as init process
[    7.497484] Root device found: initrd
modprobe: FATAL: Module r8168 not found in directory /lib/modules/5.10.120-tegra
[    7.499725] Mount initrd as rootfs and enter recovery mode
Finding OTA work dir on external storage devices
Checking whether device /dev/mmcblk?p1 exist
Device /dev/mmcblk?p1 does not exist
Checking whether device /dev/sd?1 exist
Device /dev/sd?1 does not exist
Checking whether device /dev/nvme?n1p1 exist
Looking for OTA work directory on the device(s): /dev/nvme0n1p1
mount /dev/nvme0n1p1 /mnt
[    7.523062] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null)
is_boot_only_partition /mnt
The mounted /dev/nvme0n1p1 is boot partition, try locating rootfs partition and mount it...
mount_rootfs_partition /dev/nvme0n1p1 /mnt
Found encrypted rootfs partition /dev/nvme0n1p2 through UUID(296ae97d-b3c5-4b5d-9eda-a374d0a5f3ee)
umount /mnt
unlock_encrypted_partition /dev/nvme0n1p2 dm_crypt_ota dm_crypt
is_luks_partition /dev/nvme0n1p2
[    7.597771] random: ld-linux-aarch6: uninitialized urandom read (4 bytes read)
is_unlocked /dev/nvme0n1p2 unlocked_device_name
get_uuid_for_luks_partition /dev/nvme0n1p2 luks_uuid
[    7.604002] random: ld-linux-aarch6: uninitialized urandom read (4 bytes read)
[    7.609736] random: ld-linux-aarch6: uninitialized urandom read (4 bytes read)
[    9.578212] random: crng init done
[    9.596557] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
OTA work directory /mnt/ota_work is not found on /dev/nvme0n1p1
lock_encrypted_partition dm_crypt_ota
Finding OTA work dir on internal storage device
mount /dev/mmcblk0p1 /mnt
mount: /mnt: special device /dev/mmcblk0p1 does not exist.
Failed to mount /dev/mmcblk0p1 on the /mnt
Failed to run "mount_ota_work_partition /dev/mmcblk0p1 /mnt"
OTA work directory is not found on internal and external storage devices
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.0#

Flash logs (162.5 KB)
Boot logs (179.8 KB)

The setup seems to replicate every time from what I’ve tried.

Also tried adapting steps from Attempt 1, thought using flash_t234_qspi_nvme.xml instead of flash_t234_qspi.xml would work, but it doesn’t.

Help would be greatly appreciated

7

hello _Becu,

you meant you’re using this test key? $ echo "f0e0d0c0b0a001020304050607080900" > sym2_t234.key?

you don’t need to update eks image because that’s same as default image.
please try following below, to confirm you’re given keys correctly, and running correct xml configuration file for image flashing.
for example,
$ echo "f0e0d0c0b0a001020304050607080900" > sym2_t234.key
$ sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -i ./sym2_t234.key --no-flash --showlogs --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 100GiB --append --external-only jetson-orin-nano-devkit external

8

Hello,

Yes, again, I didn’t do any modifications to the example.sh. I’m going through the entire flow, as if I’m using custom keys, to validate that the steps work.

Linux_for_Tegra$ cat sym2_t234.key 
f0e0d0c0b0a001020304050607080900

OPTEE passes the key extraction process, so I’m assuming the key and EKB are correct.

9

how about re-flash the target with above commands to check again?

10

Should I run your command first, then flash with the one that worked for me in my previous comment?
So

  1. sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -i ./sym2_t234.key --no-flash --showlogs --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 100GiB --append --external-only jetson-orin-nano-devkit external
  2. sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --external-device nvme0n1p1 -i ./sym2_t234.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --network usb0 jetson-orin-nano-devkit external

I see the one you provided has --no-flash in it

11

you may running with --no-flash to create image blob locally, and enabling --flash-only to flash the target with two step approach.
or, please exclude --no-flash from the command-line to flash the target directly.

12

The command is similar to the one I used in Attempt 4, but has --append, which is Only applicable when using with --no-flash --external-only option.. I’m not using --no-flash, so I’ll also remove the --append. The only difference remains the extra -S 100GiB then.

In Attempt 4 I managed to succesfully flash the device and boot past the key extraction process, but it failed with mount: /mnt: special device /dev/mmcblk0p1 does not exist . Do you think the issue is with the partition layout?

Ran your command, but it kept failing during the flash with

[   4.8709 ] End sector for APP_ENC, expected at: 122159070, actual: 0
[   4.8709 ] 
Error: Return value 4

as I did not modify the flash_l4t_t234_nvme_rootfs_enc.xml to accomodate the 100GiB partition.

Modified your command from 100GiB to 8GiB:
sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -i ./sym2_t234.key --showlogs --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 8GiB --external-only jetson-orin-nano-devkit external

Steps:

  1. rm -rf Linux_for_Tegra
  2. re-create Linux_for_Tegra from sources, apply binaries, create default user
  3. cd Linux_for_Tegra
  4. copy eks img and sym2_t234.key
  5. sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 -i ./sym2_t234.key --showlogs --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -S 8GiB --external-only jetson-orin-nano-devkit external

The result is the same as in Attempt 4. Device gets flashed, boots and then fails with the same error:

EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
OTA work directory /mnt/ota_work is not found on /dev/nvme0n1p1
lock_encrypted_partition dm_crypt_ota
Finding OTA work dir on internal storage device
mount /dev/mmcblk0p1 /mnt
mount: /mnt: special device /dev/mmcblk0p1 does not exist.
Failed to mount /dev/mmcblk0p1 on the /mnt
Failed to run "mount_ota_work_partition /dev/mmcblk0p1 /mnt"
OTA work directory is not found on internal and external storage devices
13

Hi, could you give some proper instructions about Attempt 3, please?

14

HI,

Check your UART logs, the setup that worked for me had the MB1, OPTEE and UEFI versions from March 2023. These are the August versions, for reference:
MB1 (version: 1.2.0.0-t234-54845784-562369e5)
OP-TEE version: 3.21 (gcc version 9.3.0 (Buildroot 2020.08)) #2 Tue Aug 1 19:39:55 UTC 2023 aarch64
Jetson UEFI firmware (version 4.1-33958178 built on 2023-08-01T19:34:02+00:00)

Follwing the steps in UEFI Secureboot updated the March versions to the August ones.

Other than that, steps are the same as the ones I referenced above in “Steps” but with the flash command from Attempt 2.

IMO now that the device boots, these versions are not the issue.

15

So, you managed to enable disk encryption with this? I mean does your work lead to the solution of your problem(and encryption at all)?

16

Yes, device booted and disk encryption was working, but it isn’t officially supported on the Orin NX (from what the docs say). Also Secureboot/ UEFI Secureboot are also not officially supported on the NX with those versions.

On second thought, the issue with the system not booting properly from Attempt 4 might be related to the versions, as there’s a log with L4TLauncher: Attempting Recovery Boot.

17

Ok, thanks for your work!

20

@JerryChang Do you have any ideas on why it might attempt to boot into recovery right on the first boot after flashing? Or any others things to try?

21

Hi @_Becu in your previous post, you mentioned “The result is the same as in Attempt 4. Device gets flashed, boots and then fails with the same error:”, did you get that resolved? a bit confusing when you mentioned “device booted and disk encryption was working”
Thanks

22

hello _Becu,

we’ve test again locally to confirm disk encryption is working.

here’re our test steps for your reference,
Preparation: $ sudo apt-get install cryptsetup.
note, it’s cryptsetup utility to create encrypted rootfs for image flashing.

  1. Generate images for QSPI:
    $ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --no-flash --network usb0 jetson-orin-nano-devkit internal
  2. Generate the key:
    $ echo "f0e0d0c0b0a001020304050607080900" > ekb.key
  3. Generate images for external storage device:
    $ sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./ekb.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --append --network usb0 jetson-orin-nano-devkit external
  4. Flash images into the both storage devices:
    $ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only

Flash success and boot up:

$ df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/crypt_root   54G  5.6G   46G  12% /
/dev/mapper/crypt_UDA   374M   14K  350M   1% /mnt/crypt_UDA
none                    7.5G     0  7.5G   0% /dev
tmpfs                   7.6G   36K  7.6G   1% /dev/shm
tmpfs                   1.6G   19M  1.5G   2% /run
tmpfs                   5.0M  4.0K  5.0M   1% /run/lock
tmpfs                   7.6G     0  7.6G   0% /sys/fs/cgroup
/dev/nvme0n1p1          371M   97M  247M  29% /boot
tmpfs                   1.6G   72K  1.6G   1% /run/user/1000
23

Hi @JerryChang. this method works if I’m using “f0e0d0c0b0a001020304050607080900” key. Another value(random) gives ERROR: fail to unlock the encrypted dev /dev/nvme0n1p2. in UART logs. Is it possible to use another key value to make it work?

P.S. Key was generated with sudo openssl rand -rand /dev/urandom -hex 16 > ekb.key