Failed to boot up after enable the full disk encryption

I enable the full disk encryption as below. It programs to my NVEM 1TB ssd. It appears program success.

ROOTFS_ENC=1 EXT_NUM_SECTORS=1887436800  ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1  -S 40GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only   jetson-xavier-nx-devkit-tx2-nx nvme0n1p1

ROOTFS_ENC=1 EXT_NUM_SECTORS=1887436800  ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --flash-only --external-device nvme0n1 -S 40GiB  -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only jetson-xavier-nx-devkit-tx2-nx nvme0n1p1

But it failed to boot up.
My understanding is it boot up to initrd in first unencrypted partition, then switch to real encrypted rootfs. The boot up log is here,
mylog.txt (106.9 KB)
may I know if I miss anything/steps?
thanks

Hi jiangpen,

Are you using the devkit or custom board for TX2?
What’s your Jetpack version in use?

It seems you didn’t run the command to prepare the image for internal eMMC.

[0001.460] E> I2C: slave not found in slaves.                                   
[0001.464] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start t.
[0001.472] E> I2C_DEV: Failed to send register address 0x00000000.              
[0001.478] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 a.
[0001.487] E> eeprom: Failed to read I2C slave device      

and EEPROM could not be recognized from I2C bus.

Hi @KevinFFF , it is custom board with TX2 NX SOM+ NVME ssd.
the jetpack version is 32.7.4.
we don’t use internal eMMC, only use external NVME. Currently it is working with NVME ssd without encryption, but now we want to enable the full disk encryption +AB scheme.

Just not much clue on this boot failure when enable encryption. This I2C message should not impact the boot up. It is confused that seems boot to kernel , then back to uboot says “Unrecognized filesystem type”
One thing I am not clear is the /etc/crypttab file, I changed according to my disk UUID, but not sure that is correct.
I believe the overall process is it boots to unencrypted APP which has initrd first, then in initrd, it reads /etc/crypttab , then boot to encrypted APP_ENC partition.
any hints are appreciate.
thanks a lot

Please refer to the following steps to flash your board with disk-encryption enabled.

a. create default key
$ echo "00000000000000000000000000000000" > sym2.key

b. create internal
$ sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --network usb0 jetson-xavier-nx-devkit-tx2-nx internal

c. create external
(you may specify 1887436800 for num_sectors in <Linux_for_Tegra>/tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml)
$ sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./sym2.key -S 100GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only --append --network usb0 jetson-xavier-nx-devkit-tx2-nx external

d. flash both
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only

Hi @KevinFFF , thanks for reply, today I tried your approach.
first thing is my version is 32.7.4, which does not support -i key option, so I get rid of it.
I changed the programming steps are below

./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash  jetson-xavier-nx-devkit-tx2-nx internal

ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1  -S 40GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only --append  jetson-xavier-nx-devkit-tx2-nx external

./tools/kernel_flash/l4t_initrd_flash.sh --showlogs  --flash-only

while I got an error of program

Writing primary_gpt partition done
Error: Invalid argument during seek for read on /dev/sdb
[ 0]: l4t_flash_from_kernel: Error: partprobe failed. This indicates that:
 -   the xml indicates the gpt is larger than the device storage
 -   the xml might be invalid
 -   the device might have a problem.
 Please make correction.

the full log is here
flash.log (90.4 KB)
the config is here
flash_l4t_nvme_rootfs_enc.xml.txt (9.1 KB)
(I changed the num_sectors=“1887436800” and 1887436800, neither worked)
may I know why still need program the internal device?
any hints are appropriate. thanks a lot

It seems you failed with the first command.

Could you try to run the following command to flash the board and share the result?

$ sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only -S 40GiB jetson-xavier-nx-devkit-tx2-nx external

@KevinFFF , thanks for reply,
I tried your command, it gives me different error.
flash_1-1.2_0_20240418-182558.log (12.9 KB)
the xml file is same as before.

thanks a lot

Disk /dev/sde: 1000GB

Are you using 1T NVMe drive connected through USB?

If so, please run the following command instead.

$ sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device sde -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only -S 40GiB jetson-xavier-nx-devkit-tx2-nx external

the NVMe is not connect through USB.
And I can do the flash A/B scheme without encryption successfully by using below command

sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1 -S 40GiB --showlogs -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_ab.xml  jetson-xavier-nx-devkit-tx2-nx nvme0n1p1
sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --external-device nvme0n1 -S 40GiB --showlogs -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_ab.xml jetson-xavier-nx-devkit-tx2-nx nvme0n1p1

Now we want to encrypt the rootfs, and it failed
thanks

I’m curious about why it shows /dev/sde rather than /dev/nvme0n1 here.
Could you share the lsblk and df -h result on your board?

Hi @KevinFFF , today I managed to flash success once, using command as below

export ROOTFS_ENC=1
./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only -S 40GiB jetson-xavier-nx-devkit-tx2-nx external
./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only -S 40GiB jetson-xavier-nx-devkit-tx2-nx external

I verified it bootup success and encrypted. Then I change to Encryption + A/B scheme.
I have couple of questions:

  1. it seems there is some state files once it runs flash, as if I try that exactly command again, it does not work (I did some other flash ), may I know if any files need to be cleaned up, so it can work as fresh flash?
  2. the data partition is mounted at /mnt/crypt_UDA via /etc/fstab, it seems this /etc/fstab is generated, not from rootfs. May I know how to change the /etc/fstab ?
  3. eventually goal is to support encryption + A/B scheme, I tried add ROOTFS_AB with ROOTFS_AB, but then my board failed to boot up.
    thanks a lot

It should be flashed successfully after the subsequent flash if you’ve ever flash it successfully. Or you can check what is generated from first command and clean them before flash.

Please share the lsblk result on your board.

If you are enabling rootfs a/b+ disk-encryption enabled, please confirm that you are using correct partition layout file, which should include both rootfs a/b and encrypted partitions.

hi @KevinFFF , I solved my boot up problem. It is the uboot detects ‘distro_bootpart’ as 2, but my initrd partition is 1, after I do below in uboot, it can boot up now. I still need solve that in build time though.

setenv distro_bootpart 1;saveenv;run nvme_boot

the last partition is mount at /mnt/crypt_UDA, and I need change the mount location. I know it is from systemd-fstab-generator, but I cannot find where to config it, may I know where is the config file locate? thanks.
my lsblk looks like below:

lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0            7:0    0    16M  1 loop  
mmcblk0        179:0    0  14.7G  0 disk  
├─mmcblk0p1    179:1    0     7G  0 part  
├─mmcblk0p2    179:2    0     7G  0 part  
├─mmcblk0p3    179:3    0     4M  0 part  
├─mmcblk0p4    179:4    0     4M  0 part  
├─mmcblk0p5    179:5    0   512K  0 part  
├─mmcblk0p6    179:6    0   512K  0 part  
├─mmcblk0p7    179:7    0   512K  0 part  
├─mmcblk0p8    179:8    0   512K  0 part  
├─mmcblk0p9    179:9    0     3M  0 part  
├─mmcblk0p10   179:10   0     3M  0 part  
├─mmcblk0p11   179:11   0     2M  0 part  
├─mmcblk0p12   179:12   0     4M  0 part  
├─mmcblk0p13   179:13   0     4M  0 part  
├─mmcblk0p14   179:14   0   604K  0 part  
├─mmcblk0p15   179:15   0   604K  0 part  
├─mmcblk0p16   179:16   0     1M  0 part  
├─mmcblk0p17   179:17   0     1M  0 part  
├─mmcblk0p18   179:18   0     2M  0 part  
├─mmcblk0p19   179:19   0     2M  0 part  
├─mmcblk0p20   179:20   0     6M  0 part  
├─mmcblk0p21   179:21   0     6M  0 part  
├─mmcblk0p22   179:22   0     2M  0 part  
├─mmcblk0p23   179:23   0   128M  0 part  
├─mmcblk0p24   179:24   0   128M  0 part  
├─mmcblk0p25   179:25   0    63M  0 part  
├─mmcblk0p26   179:26   0   512K  0 part  
├─mmcblk0p27   179:27   0   256K  0 part  
├─mmcblk0p28   179:28   0   256K  0 part  
├─mmcblk0p29   179:29   0    80M  0 part  
├─mmcblk0p30   179:30   0    80M  0 part  
├─mmcblk0p31   179:31   0   512K  0 part  
├─mmcblk0p32   259:13   0   512K  0 part  
├─mmcblk0p33   259:14   0   100M  0 part  
└─mmcblk0p34   259:15   0  61.8M  0 part  
mmcblk0boot0   179:32   0     4M  1 disk  
mmcblk0boot1   179:64   0     4M  1 disk  
mmcblk0rpmb    179:96   0     4M  0 disk  
zram0          252:0    0 479.1M  0 disk  [SWAP]
zram1          252:1    0 479.1M  0 disk  [SWAP]
zram2          252:2    0 479.1M  0 disk  [SWAP]
zram3          252:3    0 479.1M  0 disk  [SWAP]
nvme0n1        259:0    0 931.5G  0 disk  
├─nvme0n1p1    259:1    0   400M  0 part  /boot
├─nvme0n1p2    259:2    0  39.6G  0 part  
│ └─crypt_root 253:0    0  39.6G  0 crypt /
├─nvme0n1p3    259:3    0    63M  0 part  
├─nvme0n1p4    259:4    0   512K  0 part  
├─nvme0n1p5    259:5    0    64K  0 part  
├─nvme0n1p6    259:6    0    64K  0 part  
├─nvme0n1p7    259:7    0    80M  0 part  
├─nvme0n1p8    259:8    0    80M  0 part  
├─nvme0n1p9    259:9    0   512K  0 part  
├─nvme0n1p10   259:10   0   512K  0 part  
├─nvme0n1p11   259:11   0   100M  0 part  
└─nvme0n1p12   259:12   0 891.2G  0 part  
  └─crypt_UDA  253:1    0 891.2G  0 crypt /mnt/crypt_UDA

When I enabled the encryption+A/B, the boot up failure, now I just try to add one more encrypted APP partition into that, it is still boot up failure.

So I add one more APP_ENC_b partition for reserve A/B rootfs, but then failed to boot up.
flash_l4t_nvme_rootfs_enc_ab.xml.txt (9.8 KB)
failed.log (71.7 KB)
It seems it stay in initrd, and not jump to encrypted rootfs anymore.
any hints are appreciated, thanks a lot

also boot up has been hold up by systemd-fstab-generator, I think need find a way to solve this.
thanks

[   18.784745] systemd-fstab-generator[5255]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   19.218607] systemd-fstab-generator[5286]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   25.038899] Bridge firewalling registered
[   26.205923] systemd-fstab-generator[5755]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   26.588712] systemd-fstab-generator[5782]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   31.182129] systemd-fstab-generator[6074]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   31.508766] systemd-fstab-generator[6101]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   35.561242] systemd-fstab-generator[6260]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   35.888835] systemd-fstab-generator[6287]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   39.998816] systemd-fstab-generator[6444]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   40.321119] systemd-fstab-generator[6471]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   44.408896] systemd-fstab-generator[6632]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   44.739760] systemd-fstab-generator[6659]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   48.869411] systemd-fstab-generator[6816]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
[   49.209383] systemd-fstab-generator[6843]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?
...
[  128.468363] systemd-fstab-generator[10160]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?

Do you customize the partition layout file with both rootfs a/b and disk-encryption enabled?
If so, please share it for further check.

Hi @KevinFFF , thanks for reply.
Today I make a little further step, I have one success, it has A/B +encryption, but only once. I cannot reproduce it on other device, or same device again.
Basically I think it supports the A/B+ENC on TX2 NX with small changes.
But I don’t have the overall picture how boot up flow and which part causes problem. If you can share some idea that is great.

I used this command to program :

sudo ROOTFS_ENC=1 ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh  --external-device nvme0n1p1 -S 40GiB --showlogs -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc_ab.xml --external-only jetson-xavier-nx-devkit-tx2-nx external| tee log.txt

And I create a flash_l4t_nvme_rootfs_enc_ab.xml as this
flash_l4t_nvme_rootfs_enc_ab.xml.txt (9.7 KB)
so it happens to work once, and device boot up success, and I can see two rootfs and data partition are encrypted and boot up, the lsblk is as below:

root@jeteye:~# lsblk -o name,mountpoint,label,size,uuid
NAME                MOUNTPOINT LABEL   SIZE UUID
mmcblk0                               14.7G 
├─mmcblk0p1                              7G 63346e96-c59d-462e-8e3f-fc2d294d51e4
├─mmcblk0p2                              7G 87008137-01ee-4eaf-a6d4-8ac16202d7c4
├─mmcblk0p3                              4M 
├─mmcblk0p4                              4M 
├─mmcblk0p5                            512K 
├─mmcblk0p6                            512K 
├─mmcblk0p7                            512K 
├─mmcblk0p8                            512K 
├─mmcblk0p9                              3M 
├─mmcblk0p10                             3M 
├─mmcblk0p11                             2M 
├─mmcblk0p12                             4M 
├─mmcblk0p13                             4M 
├─mmcblk0p14                           604K 
├─mmcblk0p15                           604K 
├─mmcblk0p16                             1M 
├─mmcblk0p17                             1M 
├─mmcblk0p18                             2M 
├─mmcblk0p19                             2M 
├─mmcblk0p20                             6M 
├─mmcblk0p21                             6M 
├─mmcblk0p22                             2M 
├─mmcblk0p23                           128M 
├─mmcblk0p24                           128M 
├─mmcblk0p25                            63M 
├─mmcblk0p26                           512K 
├─mmcblk0p27                           256K 
├─mmcblk0p28                           256K 
├─mmcblk0p29                            80M 
├─mmcblk0p30                            80M 
├─mmcblk0p31                           512K 
├─mmcblk0p32                           512K 
├─mmcblk0p33                           100M 
└─mmcblk0p34                          61.8M 
mmcblk0boot0                             4M 
mmcblk0boot1                             4M 
mmcblk0rpmb                              4M 
nvme0n1                              931.5G 
├─nvme0n1p1         /boot              400M 1e59afbd-2839-4b42-919f-f147e840d1ff
├─nvme0n1p2                           39.6G 90b5587a-b342-4833-b23d-264b199c0a6d
│ └─crypt_root_other
│                                     39.6G 
├─nvme0n1p3                           39.6G d39ad044-f70b-40be-83b3-14be62fb4fc7
│ └─crypt_root      /                 39.6G 
├─nvme0n1p4                             63M 
├─nvme0n1p5                            512K 
├─nvme0n1p6                             64K 
├─nvme0n1p7                             64K 
├─nvme0n1p8                             80M 
├─nvme0n1p9                             80M 
├─nvme0n1p10                           512K 
├─nvme0n1p11                           512K 
├─nvme0n1p12                           100M 
└─nvme0n1p13                         851.6G 48fe677f-0336-42b6-a080-f458e9662c90
  └─crypt_UDA       /mnt/crypt       851.6G 

However, I have trouble to reproduce that again, no matter on other unit or same unit with a new NVME SSD. It appears something persisted caused problem.
the kernel panic when I program another NVME SSD:
log.txt (17.3 KB)

I got couple of questions:

  1. when I program external-only, does the internal memory still get programmed? It looks like partition table is in internal memory
  2. may I know how to debug the initrd? I tried copy and modify bootloader/l4t_initrd.img to add some logs, anyway can print out the log to console from initrd? I suspect some error from there caused boot failure.
  3. may I know when I use external-only, where is uboot locate? I think the uboot put env var into the emmc, but not sure is uboot firmware?
  4. how is the key the initrd get to decrypt rootfs? it is kind if IPC, but where is the key physically located?
    thanks a lot

Hi @KevinFFF ,
today I got further, I can boot up my unit now. But it appears to boot up to a ‘maintenance’ mode like:

[   18.210531] tegra-asoc: sound: snd_soc_register_card failed (-517)
Press Enter for maintenance
(or press Control-D to continue): 

I remove the quiet option in extlinux.conf, and here is my boot log
bootup.txt (63.8 KB)

when boot up to this stage, my /etc/rc.local does not run, and it seems my hardware driver is not init, for example I run ifconfig I cannot see my eth0 and wifi.
The good news is two rootfs and data fs are encrypted, and can boot up.
I think it looks like some problem of the intrd handover rootfs to encrypted rootfs, and it seems not run some init steps.
I can see it boots to maintenance mode, but don’t know why. The same rootfs without encryption bootup no problem.

systemctl status
jeteye
    State: maintenance
     Jobs: 0 queued
   Failed: 0 units

I also notice there is a line here in the boot up log:

systemd-fstab-generator[2326]: Failed to create unit file /run/systemd/generator/-.mount, as it already exists. Duplicate entry in /etc/fstab?

any thought?
thanks a lot

I would suggest you record the log for both pass and fail case so that you can compare them and find the difference.

Please refer to the following partition layout file for both redundant rootfs and disk-encryption enabled.
flash_l4t_nvme_rootfs_ab_enc.xml (13.3 KB)

Could you interact in console after you see this message?