Kernel panic when trying to boot from encrypted NVME

[vdoom.heretic]

Hello!
I have TX2 NX with NVME SSD.
I flushed JetPack 4.6.6 into NVME with encryption ON by this command: sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1 -S 100GiB -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc.xml --external-only jetson-xavier-nx-devkit-tx2-nx nvme0n1p1

Then changed boot order in UBoot to boot form NVME according to this doc: https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3276/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html#wwpID0E0RJ0HA

And now jetson stuck in endless reboot loop.
Log:

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x100
[    0.000000] Linux version 4.9.337-tegra (buildbrain@mobile-u64-5497-d8000) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a4
[    0.000000] Boot CPU: AArch64 Processor [411fd073]
[    0.000000] OF: fdt:memory scan node memory@80000000, reg size 80,
[    0.000000] OF: fdt: - 80000000 ,  70000000
[    0.000000] OF: fdt: - f0200000 ,  85600000
[    0.000000] OF: fdt: - 175e00000 ,  200000
[    0.000000] OF: fdt: - 176600000 ,  200000
[    0.000000] OF: fdt: - 177000000 ,  200000
[    0.000000] earlycon: uart8250 at MMIO32 0x0000000003100000 (options '')
[    0.000000] bootconsole [uart8250] enabled
[    0.000000] Found tegra_fbmem: 00800000@96088000
[    0.000000] Found lut_mem: 00002008@96085000
[    1.495145] imx219 9-0010: imx219_board_setup: error during i2c read probe (-121)
[    1.502803] imx219 9-0010: board setup failed
[    1.530771] imx219 10-0010: imx219_board_setup: error during i2c read probe (-121)
[    1.538440] imx219 10-0010: board setup failed
[    2.082520] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
[    2.082520] 
[    2.091688] CPU: 4 PID: 1 Comm: bash Not tainted 4.9.337-tegra #1
[    2.097797] Hardware name: lanai-3636 (DT)
[    2.101904] Call trace:
[    2.104363] [<        (ptrval)>] dump_backtrace+0x0/0x198
[    2.109776] [<        (ptrval)>] show_stack+0x24/0x30
[    2.114842] [<        (ptrval)>] dump_stack+0xa0/0xc4
[    2.119906] [<        (ptrval)>] panic+0x128/0x2a4
[    2.124708] [<        (ptrval)>] complete_and_exit+0x0/0x30
[    2.130294] [<        (ptrval)>] do_group_exit+0x40/0xa8
[    2.135619] [<        (ptrval)>] __wake_up_parent+0x0/0x40
[    2.141117] [<        (ptrval)>] el0_svc_naked+0x34/0x38
[    2.146443] SMP: stopping secondary CPUs
[    2.150391] Kernel Offset: disabled
[    2.153886] Memory Limit: none
[    2.156947] trusty-log panic notifier - trusty version Built: 23:39:30 Nov  4 2024 [    2.168461] Rebooting in 5 seconds..

If I reflush jetson NVME with the same parameters but without encryption it loads perfectly fine. Reflushing eMMC with encryption switched on works fine as well.

Is it possible to fix it, or maybe there is an different approach to get encrypted APP partition on NVME with TX2 NX?

[JerryChang]

hello vdoom.heretic,

did you update EKS image with your user key, (i.e. ekb.key)?
please see-also.. $OUT/Linux_for_Tegra/tools/kernel_flash/README_initrd_flash.txt

BTW,
please setup serial console, we may checking booting logs for more details.

[vdoom.heretic]

Hello @JerryChang !

I created ekb.key file with all zeroes echo "00000000000000000000000000000000" > ekb.key befor flushing.
Also I created a user sudo ./tools/l4t_create_default_user.sh -u nvidia -p nvidia -a --accept-license

[JerryChang]

hello vdoom.heretic,

let’s narrow down the issue, you may replacing EKS image for quick checking.
please check [README_initrd_flash.txt] for the sample for flashing eks partition on internal device, for instance,
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh -k eks jetson-xavier mmcblk0p1
please see-also Topic 255711 for reference.

besides, please setup serial console, we need checking booting logs for more details.

[vdoom.heretic]

Hello JarryChang!

I just did it. I add flashing logs and logs from serial console just in case if they are needed. Also I reconfigured boot loader back to boot from emmc instead of nvme to just see if this flashing was successful.

flashing_logs.txt (82.0 KB)
serial_consol_logs.txt (115.6 KB)

[JerryChang]

hello vdoom.heretic,

here’s log to flash EKS partition, but the logs has truncated, is it complete and return success?

/home/vdoom/nvidia/nvidia_sdk/JetPack_4.6.6_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/tools/kernel_flash/l4t_initrd_flash_internal.sh -k eks --usb-instance 1-1 --device-instance 0 --flash-only -k eks jetson-xavier-nx-devkit-tx2-nx mmcblk0p1
Start flashing device: 1-1, rcm instance: 0, PID: 8818
Log will be saved to Linux_for_Tegra/initrdlog/flash_1-1_0_20250409-225206.log 
Ongoing processes: 8818
Ongoing processes: 8818
Ongoing processes: 8818
Ongoing processes: 8818
Ongoing processes: 8818
Ongoing processes: 8818
Ongoing processes: 8818
Ongoing processes:

[vdoom.heretic]

Hello JerryChang!

I just rerun it again and this is just how it ends

After that Jetson restart and booting.
And looks like according to initrd logs it finished flashing successfully.
flash_1-1_0_20250410-115954.log (19.5 KB)

[JerryChang]

hello vdoom.heretic,

all right, it seems you’ve done partition update.

 [ 4]: l4t_flash_from_kernel: Successfully flash the emmc
 [ 4]: l4t_flash_from_kernel: Flashing partition eks success

could you please visit Jetson Linux R32.7.6 | NVIDIA Developer to download [Driver Package (BSP) Sources] package,
please refer to below to re-generate EKS image for verification.
$public_sources/r32.7.6/Linux_for_Tegra/source/public/atf_and_trusty/trusty/app/nvidia-sample/hwkey-agent/CA_sample/tool/gen_ekb/example.sh

[vdoom.heretic]

Hello JerryChang!

After running example.sh I got eks.img, 3 key files with all zeroes and fv_ekb file.

[JerryChang]

@vdoom.heretic

all zeroes and fv_ekb file.

actually, you should keep FV as bad66eb4484983684b992fe54a648bb8, which is default vector for EKB.

[vdoom.heretic]

Hello JerryChang!

I mean kek2_key, sym.key and sym2.key has zeroes.
fv_ekb indeed has non-zero value

[JerryChang]

hello vdoom.heretic,

please setup serial console, we’ll need complete booting logs for checking.

[vdoom.heretic]

Hello JerryChang!

Seems like after flashing EKS partition it’s now booting from encrypted NVME without problems.

Here are logs from serial cosole, just in case:
boot_to_emmc.txt (33.9 KB)
boot_to_nvme.txt (33.5 KB)

Does it mean that for TX2 NX I have to manually reflush EKS image each time I flashing JetPack on encrypted NVME? Or I just did it in the wrong way?

[JerryChang]

hello vdoom.heretic,

yes, because you’ll need to have unique ECID (per device) to enable disk encryption.

there’s method to create encrypted images with a generic key, Creating Encrypted Images with a Generic Key.
however, this was added by JP-5.1.3/r35.5.0 public release, which means TX2 NX cannot support disk encryption with a generic key.

[vdoom.heretic]

Thank you JerryChang for your help 🙏