Orin NX Disk Encryption LUKS Header Error - L4T 35.5.0

Hello,

I’m currently facing an issue with disk encryption on the Orin NX 16GB with L4T 35.5.0.

My distribution is Buildroot based, but all of the flashing scripts are from the L4T sources so I would expect it to behave the same as a Jetpack installation.

[   10.245753] Run /init as init process
[   10.262337] Root device found: UUID=51424804-e148-490a-87ec-d3a99b067f27
[   10.281267] Cryptsetup version: 2.6.1
[   10.416932] ERROR: encrypted dev /dev/nvme0n1p2 is not LUKS device.
[   10.425129] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00

I am running the following commands as per the Jetson documentation for flashing external NVME with encryption;

Put target in recovery mode and then run the following;

sudo  ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --no-flash --network usb0 jetson-orin-nano-devkit internal 2>&1 | tee flash_qspi.log

sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash  --external-device nvme0n1p1 -i ./ekb.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --append --network usb0 jetson-orin-nano-devkit external 2>&1 | tee flash_nvme.log

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only 

In the initrd, if I run cryptsetup isLuks /dev/nvme0n1p2 --verbose --debug I can see that the on-disk vs in-memory checksum is different. Is the LUKS header somehow being corrupted?

# cryptsetup 2.6.1 processing "cryptsetup isLuks /dev/nvme0n1p2 --verbose --debug"
# Verifying parameters for command isLuks.
# Running command isLuks.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /dev/nvme0n1p2.
# Trying to open and read device /dev/nvme0n1p2 with direct-io.
# Initialising device-mapper backend library.
# Trying to load any crypt type from device /dev/nvme0n1p2.
# Crypto backend (OpenSSL 1.1.1t  7 Feb 2023) initialized in cryptsetup library version 2.6.1.
# Detected kernel Linux 5.10.192-g6099b0d88488 aarch64.
# Loading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/nvme0n1p2.
# Opening lock resource file /run/cryptsetup/L_259:2
# Verifying lock handle for /dev/nvme0n1p2.
# Device /dev/nvme0n1p2 READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/nvme0n1p2
# Verifying locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:622f5c554f6204076a6e9498ce28afbcc62d05481edcdba1c8b75f0e5421c82c (on-disk)
# Checksum:ca0b6827e1349d8fca331da1c752e2282720cd611b71667a9e534511a9adacfe (in-memory)
# LUKS2 header checksum error (offset 0).
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/nvme0n1p2
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:9ac304a2d396b61251daa49d1a2a42d0e4e377bf4fd8c50ccaa27fb6a4dc68e1 (on-disk)
# Checksum:c3687a7d6acb764e3e6e5b6dd595b214500392c7d22032c0746f7cb4530dd3b7 (in-memory)
# LUKS2 header checksum error (offset 16384).
# Trying to read secondary LUKS2 header at offset 0x8000.
# Reusing open ro fd on device /dev/nvme0n1p2
# Trying to read secondary LUKS2 header at offset 0x10000.
# Reusing open ro fd on device /dev/nvme0n1p2
# Trying to read secondary LUKS2 header at offset 0x20000.
# Reusing open ro fd on device /dev/nvme0n1p2
# Trying to read secondary LUKS2 header at offset 0x40000.
# Reusing open ro fd on device /dev/nvme0n1p2
# Trying to read secondary LUKS2 header at offset 0x80000.
# Reusing open ro fd on device /dev/nvme0n1p2
# Trying to read secondary LUKS2 header at offset 0x100000.
# Reusing open ro fd on device /dev/nvme0n1p2
# Trying to read secondary LUKS2 header at offset 0x200000.
# Reusing open ro fd on device /dev/nvme0n1p2
# Trying to read secondary LUKS2 header at offset 0x400000.
# Reusing open ro fd on device /dev/nvme0n1p2
# LUKS2 header read failed (-22).
# Device /dev/nvme0n1p2 READ lock released.
# Releasing crypt device /dev/nvme0n1p2 context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/nvme0n1p2.
Command failed with code -1 (wrong or missing parameters).

Log files from flash are attached.

Is anyone able to help?

Thank you in advance.

flash_1-4.2_0_20240410-204109.log (39.6 KB)
flash_nvme.log (159.7 KB)
flash_qspi.log (213.4 KB)

Hi joe-sandom,

Are you using the devkit or custom board for Orin NX?

Could you refer to the following thread and verify with another NVMe drive?
Flash Disk encryption on external(NVME) not booting - #3 by KevinFFF

Thanks @KevinFFF , turns out there was an issue with my SSD, I tried another one but ran into another issue. The rootdev is detected as a LUKS device but cannot unlock.

[   10.352052] Run /init as init process
[   10.368672] Root device found: UUID=ce10f2b1-0546-41e9-858f-afa9d9875ca5
[   10.387555] Cryptsetup version: 2.6.1
[   10.688060] ERROR: fail to unlock the encrypted dev /dev/nvme0n1p2.
[   10.698814] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00

My ekb.key should just be the same as the sample, I have not regenerated the eks image. I simply did this step prior to flashing;

echo "f0e0d0c0b0a001020304050607080900" > ekb.key 

When debugging the init script, it looks like the following command returns error code 134 - I’m not sure what that represents in the nvluks-srv-app though.

nvluks-srv-app -u -c "${crypt_disk_uuid}" | eval ${run_cryptsetup} luksOpen "${enc_dev}" "${enc_dm_name}";

Any ideas?

Update to this, seems to be working now. I had to modify the run_cryptsetup command to include /lib/aarch64-linux-gnu in LD_LIBRARY_PATH.

run_cryptsetup="LD_LIBRARY_PATH=\"/lib/cryptsetup:/lib/aarch64-linux-gnu\" /lib/cryptsetup/ld-linux-aarch64.so.1 /sbin/cryptsetup "

Otherwise, the initrd falls over with;

libgcc_s.so.1 must be installed for pthread_cancel to work

When trying to pass the nvluks-srv-app output into run_cryptsetup.

I can see there is this conf file etc/ld.so.conf.d/aarch64-linux-gnu.confbut doesn’t look like the initrd is build with ldconfig included. Maybe I’m missing something or it’s to do with the generation of the initrd based on my Buildroot build and missing copying ldconfig into the ramfs.

1 Like

@KevinFFF I have realised that this issue is not actually solved. It just happened to work when I tried with a 1TB SSD and a 120GB SSD as opposed to the 160GB SSD I was originally using. I have also tried another brand new 160GB SSD and had the same issue.

I’m not sure if it might be to do with the number of sectors causing alignment issues, any ideas?

Here is the 120GB SSD that works;

Here is the 160GB SSD that does not work;

Are you in recovery boot state?

Do you mean both 1TB SSD and 120GB SSD work as expected but your another 160GB not work?
Please match the number of sectors read from NVMe drive as the sectors defined in partition layout file.

Yes, I’m following the exact same process between all SSD’s and I’m in recovery mode, only the 160GB SSD does not work and I have tried two of them. The 160GB SSD is https://www.innodisk.com/en/products/flash-storage/m2-pcie/m2-p80-4ig2-p and the 120GB SSD is https://www.innodisk.com/en/products/flash-storage/m2-pcie/m2-p80-3te6. The fdisk output I am grabbing is from the initrd shell once the unlocking has failed.

I’ve tried setting EXT_NUM_SECTORS to 312581808 to be the full disk size, but I have also tried setting EXT_NUM_SECTORS to around 150GB to be a bit smaller than the actual capacity e.g. EXT_NUM_SECTORS=293343480.

The behaviour was the same. The number of sectors is an even number and the sector size is 512 bytes. I’m not sure if there is some specific technology on this SSD model that is causing issues.

Any ideas?

What is your current issue with 160GB SSD? Flash failed for boot up issue?
Please share both flash log and serial console log.

Hi @KevinFFF , it is as per the original post - flash logs and snippet of serial console log with the error are provided.

The original post was in relation to the 160GB SSD and was “fixed” by changing to the 120GB SSD. Since then, I tried another 160GB SSD (same make and model as the other 160GB SSD) and I had the same problem. So it was not a faulty SSD but seems to be a systematic fault with this certain model of 160GB SSD.

Do you need further details for analysis?

If 120GB SSD could work as expected but 160GB SSD not, I would suggest you can compare the flash log and also serial console log between these 2 cases to find out the differences.

I also want to know if 120GB is not enough for your case so that you want 160GB SSD working.

Hi @KevinFFF, I have done the comparison previously - there is nothing that stands out. Recently I have performed a hexdump on the disk to show the LUKS header and can see a difference.

Good header on 120GB SSD;

00001000  7b 22 6b 65 79 73 6c 6f  74 73 22 3a 7b 22 30 22  |{"keyslots":{"0"|
00001010  3a 7b 22 74 79 70 65 22  3a 22 6c 75 6b 73 32 22  |:{"type":"luks2"|
00001020  2c 22 6b 65 79 5f 73 69  7a 65 22 3a 33 32 2c 22  |,"key_size":32,"|
00001030  61 66 22 3a 7b 22 74 79  70 65 22 3a 22 6c 75 6b  |af":{"type":"luk|
00001040  73 31 22 2c 22 73 74 72  69 70 65 73 22 3a 34 30  |s1","stripes":40|
00001050  30 30 2c 22 68 61 73 68  22 3a 22 73 68 61 32 35  |00,"hash":"sha25|
00001060  36 22 7d 2c 22 61 72 65  61 22 3a 7b 22 74 79 70  |6"},"area":{"typ|
00001070  65 22 3a 22 72 61 77 22  2c 22 6f 66 66 73 65 74  |e":"raw","offset|
00001080  22 3a 22 33 32 37 36 38  22 2c 22 73 69 7a 65 22  |":"32768","size"|
00001090  3a 22 31 33 31 30 37 32  22 2c 22 65 6e 63 72 79  |:"131072","encry|
000010a0  70 74 69 6f 6e 22 3a 22  61 65 73 2d 78 74 73 2d  |ption":"aes-xts-|
000010b0  70 6c 61 69 6e 36 34 22  2c 22 6b 65 79 5f 73 69  |plain64","key_si|
000010c0  7a 65 22 3a 33 32 7d 2c  22 6b 64 66 22 3a 7b 22  |ze":32},"kdf":{"|
000010d0  74 79 70 65 22 3a 22 61  72 67 6f 6e 32 69 22 2c  |type":"argon2i",|
000010e0  22 74 69 6d 65 22 3a 36  2c 22 6d 65 6d 6f 72 79  |"time":6,"memory|
000010f0  22 3a 31 30 34 38 35 37  36 2c 22 63 70 75 73 22  |":1048576,"cpus"|
00001100  3a 34 2c 22 73 61 6c 74  22 3a 22 41 63 51 75 31  |:4,"salt":"AcQu1|
00001110  35 70 50 34 68 6f 6f 43  2b 42 45 57 67 72 4f 5a  |5pP4hooC+BEWgrOZ|
00001120  57 58 48 35 42 4b 4a 32  6d 32 33 50 2f 76 30 4e  |WXH5BKJ2m23P/v0N|
00001130  52 50 64 4b 47 51 3d 22  7d 7d 7d 2c 22 74 6f 6b  |RPdKGQ="}}},"tok|
00001140  65 6e 73 22 3a 7b 7d 2c  22 73 65 67 6d 65 6e 74  |ens":{},"segment|
00001150  73 22 3a 7b 22 30 22 3a  7b 22 74 79 70 65 22 3a  |s":{"0":{"type":|
00001160  22 63 72 79 70 74 22 2c  22 6f 66 66 73 65 74 22  |"crypt","offset"|
00001170  3a 22 31 36 37 37 37 32  31 36 22 2c 22 73 69 7a  |:"16777216","siz|
00001180  65 22 3a 22 64 79 6e 61  6d 69 63 22 2c 22 69 76  |e":"dynamic","iv|
00001190  5f 74 77 65 61 6b 22 3a  22 30 22 2c 22 65 6e 63  |_tweak":"0","enc|
000011a0  72 79 70 74 69 6f 6e 22  3a 22 61 65 73 2d 78 74  |ryption":"aes-xt|
000011b0  73 2d 70 6c 61 69 6e 36  34 22 2c 22 73 65 63 74  |s-plain64","sect|
000011c0  6f 72 5f 73 69 7a 65 22  3a 35 31 32 7d 7d 2c 22  |or_size":512}},"|
000011d0  64 69 67 65 73 74 73 22  3a 7b 22 30 22 3a 7b 22  |digests":{"0":{"|
000011e0  74 79 70 65 22 3a 22 70  62 6b 64 66 32 22 2c 22  |type":"pbkdf2","|
000011f0  6b 65 79 73 6c 6f 74 73  22 3a 5b 22 30 22 5d 2c  |keyslots":["0"],|
00001200  22 73 65 67 6d 65 6e 74  73 22 3a 5b 22 30 22 5d  |"segments":["0"]|
00001210  2c 22 68 61 73 68 22 3a  22 73 68 61 32 35 36 22  |,"hash":"sha256"|
00001220  2c 22 69 74 65 72 61 74  69 6f 6e 73 22 3a 32 32  |,"iterations":22|
00001230  31 34 30 35 2c 22 73 61  6c 74 22 3a 22 6a 47 77  |1405,"salt":"jGw|
00001240  61 6a 74 68 46 42 76 6a  65 68 48 41 4f 38 65 59  |ajthFBvjehHAO8eY|
00001250  68 79 55 5a 30 69 76 2f  56 76 4f 78 2b 55 67 44  |hyUZ0iv/VvOx+UgD|
00001260  77 4d 58 63 67 49 57 49  3d 22 2c 22 64 69 67 65  |wMXcgIWI=","dige|
00001270  73 74 22 3a 22 6e 72 2b  4a 43 4d 46 42 71 58 6d  |st":"nr+JCMFBqXm|
00001280  4f 39 47 6b 43 30 75 76  46 31 71 57 33 6a 55 52  |O9GkC0uvF1qW3jUR|
00001290  69 2f 31 69 59 68 50 55  31 41 2f 79 50 49 4a 55  |i/1iYhPU1A/yPIJU|
000012a0  3d 22 7d 7d 2c 22 63 6f  6e 66 69 67 22 3a 7b 22  |="}},"config":{"|
000012b0  6a 73 6f 6e 5f 73 69 7a  65 22 3a 22 31 32 32 38  |json_size":"1228|
000012c0  38 22 2c 22 6b 65 79 73  6c 6f 74 73 5f 73 69 7a  |8","keyslots_siz|
000012d0  65 22 3a 22 31 36 37 34  34 34 34 38 22 7d 7d 00  |e":"16744448"}}.|
000012e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00004000

Bad header on 160GB SSD;

00001000  7b 22 6b 65 79 73 6c 6f  74 73 22 3a 7b 22 30 22  |{"keyslots":{"0"|
00001010  3a 7b 22 74 79 70 65 22  3a 22 6c 75 6b 73 32 22  |:{"type":"luks2"|
00001020  2c 22 6b 65 79 5f 73 69  7a 65 22 3a 33 32 2c 22  |,"key_size":32,"|
00001030  61 66 22 3a 7b 22 74 79  70 65 22 3a 22 6c 75 6b  |af":{"type":"luk|
00001040  73 31 22 2c 22 73 74 72  69 70 65 73 22 3a 34 30  |s1","stripes":40|
00001050  30 30 2c 22 68 61 73 68  22 3a 22 73 68 61 32 35  |00,"hash":"sha25|
00001060  36 22 7d 2c 22 61 72 65  61 22 3a 7b 22 74 79 70  |6"},"area":{"typ|
00001070  65 22 3a 22 72 61 77 22  2c 22 6f 66 66 73 65 74  |e":"raw","offset|
00001080  22 3a 22 33 32 37 36 38  22 2c 22 73 69 7a 65 22  |":"32768","size"|
00001090  3a 22 31 33 31 30 37 32  22 2c 22 65 6e 63 72 79  |:"131072","encry|
000010a0  70 74 69 6f 6e 22 3a 22  61 65 73 2d 78 74 73 2d  |ption":"aes-xts-|
000010b0  70 6c 61 69 6e 36 34 22  2c 22 6b 65 79 5f 73 69  |plain64","key_si|
000010c0  7a 65 22 3a 33 32 7d 2c  22 6b 64 66 22 3a 7b 22  |ze":32},"kdf":{"|
000010d0  74 79 70 65 22 3a 22 61  72 67 6f 6e 32 69 22 2c  |type":"argon2i",|
000010e0  22 74 69 6d 65 22 3a 36  2c 22 6d 65 6d 6f 72 79  |"time":6,"memory|
000010f0  22 3a 31 30 34 38 35 37  36 2c 22 63 70 75 73 22  |":1048576,"cpus"|
00001100  3a 34 2c 22 73 61 6c 74  22 3a 22 4e 59 77 51 78  |:4,"salt":"NYwQx|
00001110  72 56 72 6e 6b 62 34 66  63 42 67 79 78 49 55 6a  |rVrnkb4fcBgyxIUj|
00001120  53 65 66 32 43 78 53 47  6b 42 2b 50 53 56 68 6d  |Sef2CxSGkB+PSVhm|
00001130  37 55 2b 35 69 45 3d 22  7d 7d 7d 2c 22 74 6f 6b  |7U+5iE="}}},"tok|
00001140  65 6e 73 22 3a 7b 7d 2c  22 73 65 67 6d 65 6e 74  |ens":{},"segment|
00001150  73 22 3a 7b 22 30 22 3a  7b 22 74 79 70 65 22 3a  |s":{"0":{"type":|
00001160  22 63 72 79 70 74 22 2c  22 6f 66 66 73 65 74 22  |"crypt","offset"|
00001170  3a 22 31 36 37 37 37 32  31 36 22 2c 22 73 69 7a  |:"16777216","siz|
00001180  65 22 3a 22 64 79 6e 61  6d 69 63 22 2c 22 69 76  |e":"dynamic","iv|
00001190  5f 74 77 65 61 6b 22 3a  22 30 22 2c 22 65 6e 63  |_tweak":"0","enc|
000011a0  72 79 70 74 69 6f 6e 22  3a 22 61 65 73 2d 78 74  |ryption":"aes-xt|
000011b0  73 2d 70 6c 61 69 6e 36  34 22 2c 22 73 65 63 74  |s-plain64","sect|
000011c0  6f 72 5f 73 69 7a 65 22  3a 35 31 32 7d 7d 2c 22  |or_size":512}},"|
000011d0  64 69 67 65 73 74 73 22  3a 7b 22 30 22 3a 7b 22  |digests":{"0":{"|
000011e0  74 79 70 65 22 3a 22 70  62 6b 64 66 32 22 2c 22  |type":"pbkdf2","|
000011f0  6b 65 79 73 6c 6f 74 73  22 3a 5b 22 30 22 5d 2c  |keyslots":["0"],|
00001200  22 73 65 67 6d 65 6e 74  73 22 3a 5b 22 30 22 5d  |"segments":["0"]|
00001210  2c 22 68 61 73 68 22 3a  22 73 68 61 32 35 36 22  |,"hash":"sha256"|
00001220  2c 22 69 74 65 72 61 74  69 6f 6e 73 22 3a 32 34  |,"iterations":24|
00001230  34 39 39 34 2c 22 73 61  6c 74 22 3a 22 65 4e 58  |4994,"salt":"eNX|
00001240  79 6b 65 74 4f 66 46 57  44 62 55 36 4f 55 73 52  |yketOfFWDbU6OUsR|
00001250  41 2f 46 48 66 50 4f 4b  74 36 77 7a 53 43 66 43  |A/FHfPOKt6wzSCfC|
00001260  37 6b 36 52 6d 64 6e 77  3d 22 2c 22 64 69 67 65  |7k6Rmdnw=","dige|
00001270  73 74 22 3a 22 76 33 63  47 4c 74 5a 67 62 4c 6c  |st":"v3cGLtZgbLl|
00001280  67 6b 67 4e 6d 64 58 78  44 2b 75 6d 75 32 38 55  |gkgNmdXxD+umu28U|
00001290  66 45 46 68 75 38 72 75  76 36 52 7a 58 56 56 4d  |fEFhu8ruv6RzXVVM|
000012a0  3d 22 7d 7d 2c 22 63 6f  6e 66 69 67 22 3a 7b 22  |="}},"config":{"|
000012b0  6a 73 6f 6e 5f 73 69 7a  65 22 3a 22 31 32 32 38  |json_size":"1228|
000012c0  38 22 2c 22 6b 65 79 73  6c 6f 74 73 5f 73 69 7a  |8","keyslots_siz|
000012d0  65 22 3a 22 31 36 37 34  34 34 34 38 22 7d 7d 00  |e":"16744448"}}.|
000012e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001400  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00004000

You can see from memory address 00001400 to 00004000 it’s filled with “ff” as opposed to zeroes.

In l4t_flash_from_kernel.sh I can see function write_sparse_image is doing a blkdiscard on the partition prior to writing the sparse image;

function write_sparse_image
{
	# sparse file
	if [ "${host_mode}" = "0" ]; then
		echo "blkdiscard ${1}"
		if ! blkdiscard "${1}"; then
			echo "Cannot erase using blkdiscard. Write zero to partition ${1}"
			echo "dd if=/dev/zero of=${1}"
			dd if=/dev/zero of="${1}" status=progress oflag=direct
		fi
........................

I think for some reason the blkdiscard command is returning success but in reality the SSD firmware is not behaving as it should. In fact, if I comment out the blkdiscard and allow the dd command to work instead, then it works without issues on the 160GB SSD. However, dd zero takes hours to flash so that is not ideal.

I checked the nvme sysfs entry for both the 120GB SSD and 160GB SSD and write_zeroes_max_bytes seems to be 0 for the 160GB SSD - so maybe this is the issue?

M.2 (P80) 3TE6;

cat /sys/block/nvme0n1p3/queue/write_zeroes_max_bytes → result is 1048576

M.2 (P80) 4IG2-P;

cat /sys/block/nvme0n1p3/queue/write_zeroes_max_bytes → result is 0

I have gone back to the SSD vendor to ask the question. If anyone on this forum has any ideas, please let me know!

Do you mean that you run dd command (dd if=/dev/zero of=“${1}” status=progress oflag=direct) to write zeros to160GB SSD and there’s no issue about “ERROR: encrypted dev /dev/nvme0n1p2 is not LUKS device.”?

It sounds more like the SSD specific issue for us.
You can provide above results to SSD vendor for further information.

Yes, that’s correct. I have found a workaround for this issue though;

If I add the --zeroout argument to blkdiscard it will force the zeroise of the partition and this works quicker than a dd zeroise.

I’ve also found out from Innodisk that the 3TE6 fills a “no data-written area” with zeroes whereas for the 4IG2-P the “no data-written area” is filled with 0xFF.

I’m not sure why this is the case but I have asked them.

The workaround described solves this issue though.

1 Like