Mass flash an encrypted file system on external storage

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
,
1

Hi!

I am currently trying to build a secure environment to flash Jetson Xavier NX modules that I could give to an external manufacturer that assembles our system. We have a custom carrier board with the production NX modules and a 128 GB NVMe attached.

I need the root file system mounted on the NVMe during booting (because the eMMC storage of the module is not enough) and I also want it to be encrypted. So if someone removes the NVMe from the board once the system is flashed, the information inside it is not accessible. I am using L4T 32.7.1 and I am not considering changing at the moment.

This post helps me with the commands that allow me to generate an encrypted image for the NVMe. It also generates a massflash environment that can be shared. However, when I tried to flash a different board than the one I used to generate the images with the first two commands, the flashing process looks like it finishes without problem but then the board never boots.

My steps are:

  1. Connect Board 1 to the Host PC

  2. Put board 1 in Recovery Mode and run the following command to generate the image for internal memory:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash jetson-xavier-nx-devkit-qspi internal
  1. Put board 1 in Recovery Mode again and run the following command to generate the encrypted image for NVMe:
sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -S 118GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc128.xml --external-only --massflash 1 --append jetson-xavier-nx-devkit-emmc external

The file flash_l4t_nvme_rootfs_enc128.xml is a modified version of flash_l4t_nvme_rootfs_enc.xml provided as example in Linux_for_Tegra/ folder, where I changed the num_sectors to 249980518.

  1. Put board 1 in Recovery Mode again and flash it:
sudo ./mfi_jetson-xavier-nx-devkit-emmc/tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 1 --showlogs

After the process ends, the board boots and I can login without problem. Then, I connect board 2, put it in Recovery Mode and run the same command to flash it. The process looks like it ends without problem, however the board 2 never boots. If I use the Serial port to see the logs I see that it stops in the following line:

[    4.929281] SMP: stopping secondary CPUs
[    4.929409] Kernel Offset: disabled
[    4.929680] Memory Limit: none
[    4.929935] trusty-log panic notifier - trusty version Built: 08:57:16 Feb 19 2022 [    4.938010] Rebooting in 5 seconds..

The full log from the serial port is here:
serial_log.txt (38.3 KB)

Any help on this would be appreciated!

3

Hi jic,

[    2.157428] imx219 9-0010: imx219_board_setup: error during i2c read probe (-121)
[    2.157724] imx219 9-0010: board setup failed
[    2.181589] imx219 10-0010: imx219_board_setup: error during i2c read probe (-121)
[    2.181857] imx219 10-0010: board setup failed
\FF\E2WARNING: at platform/drivers/pg/pg-gpu-t194.c:185
WARNING: at platform/drivers/pg/pg-gpu-t194.c:185
\FF\E1[    4.927911] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00

It seems there’s kernel panic when you boot up on board 2.
Could you also check the serial console of board 1 if there’s imx219 10-0010: board setup failed?

Please also share the result of the following command on the board 1 and board 2.

$ cat /etc/nv_boot_control.conf
4

Hi @KevinFFF, thanks for your reply.

The content of /etc/nv_boot_control.conf in board 1 is:

TNSPEC 3668-301-0001-J.0-1-2-jetson-xavier-nx-devkit-emmc-internal
COMPATIBLE_SPEC 3668-301---1--jetson-xavier-nx-devkit-emmc-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

As board 2 doesn’t boot I can’t see the content of the file there.

I am attaching the log from the serial port on board 1. I can see the imx219 10-0010: board setup failed? there.

serial_log_board1.txt (38.4 KB)

5

Please reflash the board 2 w/o disk-encryption and get the result to check if the board 1 and board 2 are the same.

So, they are not relating to your reboot issue.
It seems caused from the following message.

[    4.929935] trusty-log panic notifier - trusty version Built: 08:57:16 Feb 19 2022

You didn’t add -i ./XXX.key in Step2 to generate the encrypted image for external device.
Please check if the following thread could help for your case.
Why the initrd_flash can encrypt the disk without SBKPKC mode - #20 by lhoang

6

Hi @KevinFFF , this is the content of the /etc/nv_boot_control.conf in the 2nd board:

TNSPEC 3668-301-0001-G.0-1-2-jetson-xavier-nx-devkit-emmc-internal
COMPATIBLE_SPEC 3668-301---1--jetson-xavier-nx-devkit-emmc-
TEGRA_CHIPID 0x19
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

The only difference that I can see is that it says G instead of J in the first line. I tried flashing a third board that I had, the nv_boot_control.conf is exactly the same as the first board and it didn’t boot either.
Also I tried to flash the boards including the -i argument with the following command:

sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -S 118GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc128.xml --external-only -p "-i ./sym2.key" --massflash 1 --append jetson-xavier-nx-devkit-emmc external

As I understood from the linked post, the script given, that uses the gen_ekb.py, is only needed if you activate secure boot, which we are not using, so I copied the sym2.key used from that script, without using gen_ekb.py. This didn’t work either.
In that post Ihoang also mentions that even using the key, the passphrase generated will still be per-device, I think that the problem comes from that, since we are using an image encrypted with an ECID on multiple boards with different ECIDs. Doing a bit more research and reading the gen_luks_passphrase.py file it seems that there is an option to get a generic passphrase, but I don’t know how to pass those parameters to that file within the l4t_initrd_flash.sh script. Do you know if that option will help us? And if that is the case, how do I pass that parameter? Thanks.

7

Please refer to optee/samples/hwkey-agent/host/tool/gen_ekb/example.sh for default key.
Disk Encryption — Jetson Linux Developer Guide documentation (nvidia.com)

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.