Orin Nano 8GB Encrypted SSD flashing FAILS on prod-module - devboard is OK

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
, , , ,
1

Hi Team,

We need to encrypt the partition on the production device with Jetson Orin Nano 8Gb module and custom carrier board.

First of all I found a solution for Jetson Orin Nano 8Gb devboard.
It’s based on these two topics:

Generate custom key

cd ./source/public/optee/samples/hwkey-agent/host/tool/gen_ekb/
./example.sh
cd ../../../../../../../..

cp ./source/public/optee/samples/hwkey-agent/host/tool/gen_ekb/sym2_t234.key ./sym2_t234.key
rm ./bootloader/eks_t234.img
cp ./source/public/optee/samples/hwkey-agent/host/tool/gen_ekb/eks_t234.img ./bootloader/eks_t234.img

Modify
./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml

like this for ~230Gb device and 200Gb LUKS partition

   <device type="external" instance="0" sector_size="512" num_sectors="460000000" >   
...
        <partition name="APP_ENC" id="2" type="data" encrypted="true" reencrypt="false">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 214748364800 </size>

Connect dev-board by USB and UART to host machine.
Put it to recovery mode.

Explicitly build something with new key by outdated scripts

sudo ./flash.sh --no-flash -k A_eks -i "sym2_t234.key" jetson-agx-orin-devkit mmcblk0p1

Prepare images for internal device (and reset flashing image folders)

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c ./bootloader/generic/cfg/flash_t234_qspi_nvme.xml" --no-flash --network usb0 jetson-orin-nano-devkit internal

Copy something key-related manually to internal device

sudo cp ./bootloader/eks_t234_sigheader.img.encrypt ./tools/kernel_flash/images/internal/eks_t234_sigheader.img.encrypt

Prepare images for external device

sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./sym2_t234.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --append --network usb0 jetson-orin-nano-devkit external

Flash to device physically

sudo systemctl stop udisks2.service
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only

This sequence is working and I have my dev-board flashed with an encrypted partition.

Successul flashing log flash_5-1_0_20240702-170918.log.txt (49.8 KB)

Also I attach the part of UART output before and after “waiting for reboot phase” flashing_uart_devboard_success.log (7.1 KB)



PRODUCTION BOARD HAS AN ISSUE

First of all I need to use the first stage with a BOARDSKU explicity mentioned

sudo BOARDID=3767 BOARDSKU=0005 ./flash.sh --no-flash -k A_eks -i "sym2_t234.key" jetson-agx-orin-devkit mmcblk0p1

This SKU is different that production board has.
Dev-board BOARDSKU=0005
Prod-board BOARDSKU=0003

Image building stages passed well.

Log: build_internal_prodboard.log (190.0 KB)

Log: build_external_prodboard.log (134.5 KB)

Flashing process stuck on waiting for reboot and failed on timeout even hasn’t started the real flashing
Log:flashing_host_prodboar_fail.log (9.6 KB)

The last part of UART output for failed prod-board flashing is
flashing_uart_prodboard_fail.log (6.8 KB)

[    6.758618] tegra-qspi 3270000.spi: Adding to iommu group 8
[    6.759258] tegra-qspi 3270000.spi: Prod config not found for QSPI: -19
[    6.760314] spi-nor spi0.0: mx25u51279g (65536 Kbytes)
[   67.859091] using random self ethernet address
[   67.859097] using random host ethernet address
[   67.866284] Mass Storage Function, version: 2009/09/11
[   67.866289] LUN: removable file: (no medium)
[   67.867760] LUN: removable file: (no medium)
Add /dev/nvme0n1
[   67.868994] LUN: removable file: (no medium)
[   67.870165] LUN: removable file: (no medium)
[   67.872963] usb0: HOST MAC aa:0d:b6:4f:96:37
[   67.872968] usb0: MAC 9a:5f:79:9e:ca:f8
[   67.875135] tegra-xudc 3550000.usb: EP 0 (type: ctrl, dir: out) enabled
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.1#

Actually I have the same Error -19 for QSPI and bash-5.1# during the successful flashing process for dev-board, so it’s not looking as an issue

[    6.655278] tegra-qspi 3270000.spi: Adding to iommu group 9
[    6.655796] tegra-qspi 3270000.spi: Prod config not found for QSPI: -19
[    6.656846] spi-nor spi0.0: mx25u51279g (65536 Kbytes)
[   67.756256] using random self ethernet address
[   67.756262] using random host ethernet address
[   67.763332] Mass Storage Function, version: 2009/09/11
[   67.763338] LUN: removable file: (no medium)
[   67.764679] LUN: removable file: (no medium)
Add /dev/nvme0n1
[   67.766039] LUN: removable file: (no medium)
[   67.767218] LUN: removable file: (no medium)
[   67.769904] usb0: HOST MAC 12:89:79:64:44:f7
[   67.769910] usb0: MAC 96:23:8e:71:d7:34
[   67.771123] tegra-xudc 3550000.usb: EP 0 (type: ctrl, dir: out) enabled
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.1# [   68.319702] tegra-xudc 3550000.usb: EP 5 (type: intr, dir: in) enabled
[   68.319724] tegra-xudc 3550000.usb: EP 3 (type: bulk, dir: in) enabled
[   68.319737] tegra-xudc 3550000.usb: EP 2 (type: bulk, dir: out) enabled
[   68.319860] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
[   68.320014] tegra-xudc 3550000.usb: EP 7 (type: bulk, dir: in) enabled
[   68.320031] tegra-xudc 3550000.usb: EP 4 (type: bulk, dir: out) enabled

Any suggestions how to fix this flashing process for production Jetson Orin Nano 8Gb board with QSPI+SSD by file-system with Encrypted partition?

I use Jetpack 6 for this

wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v2.0/release/jetson_linux_r36.2.0_aarch64.tbz2
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v2.0/release/tegra_linux_sample-root-filesystem_r36.2.0_aarch64.tbz2
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v2.0/sources/public_sources.tbz2
2

Just received different message in the UART output

[   11.446144] tegra-qspi 3270000.spi: Adding to iommu group 8
[   11.446769] tegra-qspi 3270000.spi: Prod config not found for QSPI: -19
[   11.447833] spi-nor spi0.0: mx25u51279g (65536 Kbytes)
Connection timeout: device /dev/nvme0n1 is still not ready.
[   82.561447] using random self ethernet address
[   82.561452] using random host ethernet address
[   82.568679] Mass Storage Function, version: 2009/09/11
[   82.568683] LUN: removable file: (no medium)
Connection timeout: device /dev/nvme0n1 is still not ready.
[   82.570069] LUN: removable file: (no medium)
[   82.571266] LUN: removable file: (no medium)
[   82.572450] LUN: removable file: (no medium)
[   82.575347] usb0: HOST MAC c6:9a:3c:b7:ae:51
[   82.575353] usb0: MAC d2:da:e3:8a:7a:fe
[   82.577542] tegra-xudc 3550000.usb: EP 0 (type: ctrl, dir: out) enabled
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

Now I tried to prepare images for internal flash like this:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c ./bootloader/generic/cfg/flash_t234_qspi.xml --no-systemimg" --no-flash --network usb0 jetson-orin-nano-devkit internal

It was my variation on the Workflow 4 from README_initrd_flash.txt

Workflow 4: How to flash to device with internal QSPI and an external storage device:
Some Jetson devices like Jetson Orin NX and Jetson Xavier NX have an internal QSPI and an external
storage device, which flash.sh may not have support flashing yet. In this case you can use the
following commands:

For a device with internal QSPI and external NVMe:
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 \
      -c tools/kernel_flash/flash_l4t_external.xml \
      -p "-c bootloader/generic/cfg/flash_t234_qspi.xml --no-systemimg" --network usb0 \
      <board> external
4

Hi isapient,

Normally, you should check the serial console log at this moment since the flash script is waiting for your board boot up to get ready.

It seems the whole workflow is correct since you’ve verified it working on the devkit board.

Do you have EEPROM on your custom board as well?
If not, please refer to EEPROM Modifications for your custom board.

5

Hi KevinFFF,

I’m working together with isapient on this. We have an EEPROM on the board. I have attached a custom circuit.

image (1)
image (1)1177×625 38.2 KB

I’m not sure if it is an important detail but standard flashs through sdkmanager are working on the unit.

On our board we had some issues in flashing when we didn’t add SKIP_EEPROM_CHECK=1 - it failed reading the boardid.

Normally, you should check the serial console log at this moment since the flash script is waiting for your board boot up to get ready.

Please check the file flashing_uart_proadboard_fail.log that was attached by isapient.

6

So, it seems the flash script can not read the BOARD MODEL/SKU/FAB information from the EEPROM.
Could you share the result of $ cat /etc/nv_boot_control.conf on your board?

Yes, I saw this. But I want to check the full serial console to do further check.

7

I found the critical difference between prod-module and dev-board

I had on dev-board BIOS 36.2 and corresponding version of the packages.

On the production module was BIOS 36.3 and it was probably incompatible when I tried to flash it using Jetpack containing 36.2

We really tried to flash both boards using Jetpack with 36.3 packages and now it can be flashed successfully but encrypted partition can’t be opened:

[   14.897807] ERROR: fail to unlock the encrypted dev /dev/nvme0n1p2.
[   14.900334] Kernel panic - not syncing:

I suspect I setup oem_k1.key in wrong way

Original script

# [T234 example]
# Fill your OEM_K1 fuse key value
# echo "0000000000000000000000000000000000000000000000000000000000000000" > oem_k1.key

# Generate user-defined symmetric key files
# A random generate key is recommended for production, and a specified key is recommended for testing
# For each key, there are reference examples for generating random key and specifying keys.
# openssl rand -rand /dev/urandom -hex 32 > sym_t234.key    # kernel/kernel-dtb encryption key
# echo "0000000000000000000000000000000000000000000000000000000000000000" > sym_t234.key
# openssl rand -rand /dev/urandom -hex 16 > sym2_t234.key   # disk encryption key
# echo "00000000000000000000000000000000" > sym2_t234.key
# openssl rand -rand /dev/urandom -hex 16 > auth_t234.key   # uefi variables authentication key
# echo "00000000000000000000000000000000" > auth_t234.key

python3 gen_ekb.py -chip t234 -oem_k1_key oem_k1.key \
        -in_sym_key sym_t234.key \
        -in_sym_key2 sym2_t234.key \
        -in_auth_key auth_t234.key \
        -out eks_t234.img

My example.sh

# [T234 example]
# Fill your OEM_K1 fuse key value
echo "2d4a614e645267556b58703273357638792f423f4428472b4b6250655368566d" > oem_k1.key

# Generate user-defined symmetric key files
# A random generate key is recommended for production, and a specified key is recommended for testing
# For each key, there are reference examples for generating random key and specifying keys.
openssl rand -rand /dev/urandom -hex 32 > sym_t234.key    # kernel/kernel-dtb encryption key
# echo "0000000000000000000000000000000000000000000000000000000000000000" > sym_t234.key
openssl rand -rand /dev/urandom -hex 16 > sym2_t234.key   # disk encryption key
# echo "00000000000000000000000000000000" > sym2_t234.key
openssl rand -rand /dev/urandom -hex 16 > auth_t234.key   # uefi variables authentication key
# echo "00000000000000000000000000000000" > auth_t234.key

python3 gen_ekb.py -chip t234 -oem_k1_key oem_k1.key \
        -in_sym_key sym_t234.key \
        -in_sym_key2 sym2_t234.key \
        -in_auth_key auth_t234.key \
        -out eks_t234.img
8

Changing oem_k1.key back to zeroes allows me to flash dev-board using 36.3

Doing absolutely the same on production board I have an error during flashing phase
uart_prodboar_36_3.log (57.4 KB)

As far as I understand nvme can’t be mounted

[   10.321772] tegra-qspi 3270000.spi: Adding to iommu group 8
[   10.322430] tegra-qspi 3270000.spi: Prod config not found for QSPI: -19
[   10.324011] spi-nor spi0.0: mx25u51279g (65536 Kbytes)
Connection timeout: device /dev/nvme0n1 is still not ready.
[   21.357989] using random self ethernet address
[   21.357995] using random host ethernet address
[   21.374511] tegra-xusb 3610000.usb: Firmware timestamp: 2023-02-10 03:48:10 UTC
[   21.530885] usb0: HOST MAC 8a:4f:2d:59:75:7b
[   21.530889] usb0: MAC 9e:7d:26:15:b4:18
[   21.532684] tegra-xudc 3550000.usb: EP 0 (type: ctrl, dir: out) enabled
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.1# [   22.067828] tegra-xudc 3550000.usb: EP 5 (type: intr, dir: in) enabled
[   22.067849] tegra-xudc 3550000.usb: EP 3 (type: bulk, dir: in) enabled
[   22.067862] tegra-xudc 3550000.usb: EP 2 (type: bulk, dir: out) enabled
[   22.067978] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
[   97.252083] NFS: state manager: check lease failed on NFSv4 server fc00:1:1:0::1 with error 13
[   98.272811] NFS: state manager: check lease failed on NFSv4 server fc00:1:1:0::1 with error 13

That leads to stop flashing

Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for device to expose ssh ......Waiting for device to expose ssh ...Run command: flash on fc00:1:1:0::2
SSH ready
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
[ 0]: l4t_flash_from_kernel: Serial Number: 1425123232347
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /mnt/internal/flash.idx
Number of lines is 61
max_index=60
[ 1]: l4t_flash_from_kernel: Successfully create gpt for emmc
[ 1]: l4t_flash_from_kernel: Starting to create gpt for external device
Active index file is /mnt/external/flash.idx
Number of lines is 19
max_index=18
writing item=1, 9:0:primary_gpt, 512, 19968, gpt_primary_9_0.bin, 16896, fixed-<reserved>-0, fc03ec5a55100f79887cc24e61a3543ba8aa79da
Error: Could not stat device /dev/nvme0n1 - No such file or directory.
Flash failure
Either the device cannot mount the NFS server on the host or a flash command has failed. Debug log saved to /tmp/tmp.tetZzR8b6h. You can access the target's terminal through "sshpass -p root ssh root@fc00:1:1:0::2" 
Cleaning up...
9

Please use R36.3 instead of R36.2.
It seems you can flash devkit w/o hitting the issue.
Could you try to compare the log on your board and the devkit and share the difference?

Please also share the result of cat /etc/nv_boot_control.conf on your board.

10

@KevinFFF thank you for your support and attention.

At the moment I have no access to production device and its logs as @naxty using it for other tasks.

We have found already some elegant solution looking like rough hack. The main idea is using Linux_for_Tegra folder created by SDK manager.

I’m going to double check it and then publish here the full sequence.

11

Yes, you can use the BSP package downloaded from SDK Manager.
Please just customize the board config for your custom carrier board.

12

Working solution for us, tested on Ubuntu 20.04 host machine and Ubuntu 20.04 Virtual Machine under Windows.

ENVIRONMENT SETUP

(1) Non-encrypted flashing

Install SDK manager and flash Jetson.

It creates: ~/nvidia/nvidia_sdk/JetPack_6.0_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra

(2) Install crypto-packages

Please use exactly the same version of public sources; 36.3 for me
You can check Linux_for_Tegra/kernel/*.debfor a version

cd ~/nvidia/nvidia_sdk/JetPack_6.0_Linux_JETSON_ORIN_NANO_TARGETS
wget https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v3.0/sources/public_sources.tbz2
tar -xvf public_sources.tbz2
cd Linux_for_Tegra
tar xvf source/nvidia-jetson-optee-source.tbz2
cp -r optee/samples/hwkey-agent/host/tool/gen_ekb ./

sudo apt-get update

sudo apt-get install cryptsetup dislocker libcryptsetup-dev libcryptsetup12 cryptmount qemu-user-static python3-pip

pip install cryptography pycrypto

PATCHING

(3) Generate crypto-stuff

Uncomment 4 key-generation lines in gen_ekb/example.sh

# [T234 example]
# Fill your OEM_K1 fuse key value
echo "0000000000000000000000000000000000000000000000000000000000000000" > oem_k1.key

# Generate user-defined symmetric key files
# A random generate key is recommended for production, and a specified key is recommended for testing
# For each key, there are reference examples for generating random key and specifying keys.
openssl rand -rand /dev/urandom -hex 32 > sym_t234.key    # kernel/kernel-dtb encryption key
# echo "0000000000000000000000000000000000000000000000000000000000000000" > sym_t234.key
openssl rand -rand /dev/urandom -hex 16 > sym2_t234.key   # disk encryption key
# echo "00000000000000000000000000000000" > sym2_t234.key
openssl rand -rand /dev/urandom -hex 16 > auth_t234.key   # uefi variables authentication key
# echo "00000000000000000000000000000000" > auth_t234.key

python3 gen_ekb.py -chip t234 -oem_k1_key oem_k1.key \
        -in_sym_key sym_t234.key \
        -in_sym_key2 sym2_t234.key \
        -in_auth_key auth_t234.key \
        -out eks_t234.img

Generate sym2_t234.key and eks_t234.img and apply

cd gen_ekb
./example.sh
cp sym2_t234.key ../sym2_t234.key
cp eks_t234.img ../bootloader/eks_t234.img
cd ..

(4) Tune the disk and partition

  • Run on flashed Jetson this command sudo blockdev --getsz /dev/nvme0n1
    Output is size of SSD in sectors = 468862128
  • As recommended I use smaller value: 468846000 sectors = 228928.71 MiB
  • All partitions excluding encrypted totally take a bit less than 1890 MiB
  • Let’s assign Encrypted Partition size = 227038 Mib = 238066597888 bytes

Edit tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml and replace:

  • EXT_NUM_SECTORS by 468846000
  • APP_ENC_SIZE by 238066597888
    <device type="external" instance="0" sector_size="512" num_sectors="468846000" >   
...
        <partition name="APP_ENC" id="2" type="data" encrypted="true" reencrypt="false">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 238066597888 </size>

Example for WD Green SN350 NVMe SSD 480GB:

  • Real sectors count 937703088, use 937703000 sectors = 457862.79 MiB

  • Use partition size 457862.79 MiB - 1890 MiB = 478122124247 bytes

Values for config: EXT_NUM_SECTORS = 937703000, APP_ENC_SIZE = 478122124247

FLASHING

(5) Recovery mode

Connect Jetson by USBC and UART cables and switch it to Forced Recovery mode

[black] Cable GND → GND pin 7
[green] Cable RX → TXD pin 4
[white] Cable TX → RXD pin 3

sudo minicom -D /dev/ttyUSB0 -8 -b 115200
sudo reboot --force forced-recovery

You can shorten RECOVERY MODE pin 9 and pin 10 instead of using sudo reboot

(6) Pre-config

sudo ./tools/l4t_create_default_user.sh -u user -p password -n hostname --accept-license -a
# -a for autologin

(7) Generate disk images

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs -p "-c ./bootloader/generic/cfg/flash_t234_qspi.xml" --no-flash --network usb0 jetson-orin-nano-devkit internal

sudo ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -i ./sym2_t234.key -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml --external-only --append --network usb0 jetson-orin-nano-devkit external

(8) Physical Flashing

sudo systemctl stop udisks2.service
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only

FINALIZE

(9) Adjust filesystem

Log in to Jetson using UART or Ethernet

sudo resize2fs /dev/mapper/crypt_root

(10) Double chek

Log in to Jetson using UART console and run this command on the flashed Jetson

sudo lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT
df -h

Example output for 480GB SSD

image
image431×556 36.1 KB

2 Likes
13

Surpisingly I need to Encrypt the rootfs for JetPack 5.1.2 as well.
If I do the same I’ve got

    7.281377] Mount initrd as rootfs and enter recovery mode
Finding OTA work dir on external storage devices
Checking whether device /dev/mmcblk?p1 exist
Device /dev/mmcblk?p1 does not exist
Checking whether device /dev/sd?1 exist
Device /dev/sd?1 does not exist
Checking whether device /dev/nvme?n1p1 exist
Looking for OTA work directory on the device(s): /dev/nvme0n1p1
mount /dev/nvme0n1p1 /mnt
[    7.303333] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null)
is_boot_only_partition /mnt
The mounted /dev/nvme0n1p1 is boot partition, try locating rootfs partition and mount it...
mount_rootfs_partition /dev/nvme0n1p1 /mnt
Found encrypted rootfs partition /dev/nvme0n1p2 through UUID(2d4e885a-38cb-41ce-b117-9eb84bcb76fe)
umount /mnt
unlock_encrypted_partition /dev/nvme0n1p2 dm_crypt_ota dm_crypt
is_luks_partition /dev/nvme0n1p2
[    7.364701] random: ld-linux-aarch6: uninitialized urandom read (4 bytes read)
is_unlocked /dev/nvme0n1p2 unlocked_device_name
get_uuid_for_luks_partition /dev/nvme0n1p2 luks_uuid
[    7.372101] random: ld-linux-aarch6: uninitialized urandom read (4 bytes read)
[    7.379682] random: ld-linux-aarch6: uninitialized urandom read (4 bytes read)
[    7.905114] mmc1: SDHCI controller on 3400000.sdhci [3400000.sdhci] using ADMA 64-bit
No key available with this passphrase.
Failed to unlock the LUKS partition /dev/nvme0n1p2(UUID=2d4e885a-38cb-41ce-b117-9eb84bcb76fe)
Failed to run "unlock_encrypted_partition /dev/nvme0n1p2 dm_crypt_ota dm_crypt"
Failed to run "moutn_rootfs_partition /dev/nvme0n1p1 /mnt"
Failed to run "mount_ota_work_partition /dev/nvme0n1p1 /mnt"
Finding OTA work dir on internal storage device
mount /dev/mmcblk0p1 /mnt
mount: /mnt: special device /dev/mmcblk0p1 does not exist.
Failed to mount /dev/mmcblk0p1 on the /mnt
Failed to run "mount_ota_work_partition /dev/mmcblk0p1 /mnt"
OTA work directory is not found on internal and external storage devices
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.0# [ 1080.257916] random: crng init done
14

I tried to use flash_t234_qspi_nvme.xml for internal QSPI flashing

In this case I got just fail to unlock issue

Flashing log is successful
flash_5-1_0_20240707-152725.log (39.6 KB)

UART log is not

[    9.950057] random: ld-linux-aarch6: uninitialized urandom read (4 bytes read)
[    9.963521] random: ld-linux-aarch6: uninitialized urandom read (4 bytes read)
[   10.310232] mmc1: SDHCI controller on 3400000.sdhci [3400000.sdhci] using ADMA 64-bit
[   12.457784] ERROR: fail to unlock the encrypted dev /dev/nvme0n1p2.
[   12.466868] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
[   12.477347] CPU: 5 PID: 1 Comm: bash Not tainted 5.10.120-tegra #1
[   12.484772] Hardware name: Unknown NVIDIA Orin Nano Developer Kit/NVIDIA Orin Nano Developer Kit, BIOS 4.1-33958178 08/01/2023
[   12.496497] Call trace:
[   12.499028]  dump_backtrace+0x0/0x1d0
[   12.502790]  show_stack+0x30/0x40
[   12.506194]  dump_stack+0xd8/0x138
[   12.509685]  panic+0x17c/0x384
[   12.512823]  do_exit+0xaa8/0xab0
[   12.516139]  do_group_exit+0x4c/0xb0
[   12.519807]  __arm64_sys_exit_group+0x28/0x30
[   12.524284]  el0_svc_common.constprop.0+0x80/0x1d0
[   12.529200]  do_el0_svc+0x38/0xb0
[   12.532601]  el0_svc+0x1c/0x30
[   12.535740]  el0_sync_handler+0xa8/0xb0
[   12.539673]  el0_sync+0x16c/0x180
[   12.543077] SMP: stopping secondary CPUs
[   12.547294] Kernel Offset: 0x5e5e7e9c0000 from 0xffff800010000000
[   12.553545] PHYS_OFFSET: 0xffff817400000000
[   12.557843] CPU features: 0x08040006,4a80aa38
[   12.562320] Memory Limit: none
[   12.565459] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 ]---
15

Also I recorded full UART logs and host logs while I flash Jetpack 5.1.2 using my guide linked above

minicom.log (162.3 KB)

Also device keep doing something if I don’t touch it
minicom_2.log (168.6 KB)

flash_5-1_0_20240707-162722.log (39.7 KB)

Successful SDK manager flashing logs
flash_5-1_0_20240707-155326.log (41.1 KB)

16

Also, I’d better to mention my board specifics.

In fact, I got a fresh Jetson Orin Nano 8GB dev-board, then update it to Jetpack 6 with SSD encrypted, then downgrade it back to 5.1.2 and now I’m working with it.

When I’m using SDK manager, Linux kernel flashing normally, but other packages fail: sometimes all of them, sometimes about 20% - there is no pattern, each of them could be installed or not.

Logs of my SDK manager 5.1.2 flashing:
SDKM_logs_JetPack_5.1.2_Linux_for_Jetson_Orin_Nano_modules_2024-07-08_09-07-21.zip (585.6 KB)

UART logs:
minicom_sdkmanager_jp512.log (186.2 KB)

image
image1044×720 128 KB

If I flash only Jetson Linux - no issues

image
image1048×704 116 KB

Host logs:
SDKM_logs_JetPack_5.1.2_Linux_for_Jetson_Orin_Nano_modules_2024-07-08_10-17-34.zip (511.0 KB)
Uart logs:
minicom_skdmanager_02_short.log (187.4 KB)

17

Could anyone explain me what does --generic-passphrase mean?

I’ve just tried to do following being inspired by [Security][Disk Encryption] Creating Encrypted Images with a Generic Key

Prepare keys

# [T234 example]
# Fill your OEM_K1 fuse key value
echo "0000000000000000000000000000000000000000000000000000000000000000" > oem_k1.key

# Generate user-defined symmetric key files
# A random generate key is recommended for production, and a specified key is recommended for testing
# For each key, there are reference examples for generating random key and specifying keys.
openssl rand -rand /dev/urandom -hex 32 > sym_t234.key    # kernel/kernel-dtb encryption key
openssl rand -rand /dev/urandom -hex 16 > sym2_t234.key   # disk encryption key
openssl rand -rand /dev/urandom -hex 16 > auth_t234.key   # uefi variables authentication key

python3 gen_ekb.py -chip t234 -oem_k1_key oem_k1.key \
        -in_sym_key sym_t234.key \
        -in_sym_key2 sym2_t234.key \
        -in_auth_key auth_t234.key \
        -out eks_t234.img

Generate and copy sym2_t234.key and eks_t234.img

cd gen_ekb
./example.sh

cp sym2_t234.key ../sym2_t234.key

rm ../bootloader/eks_t234.img
rm -f ../bootloader/eks_t234_sigheader.img.encrypt
cp eks_t234.img ../bootloader/eks_t234.img

cd ..

Tuned EXT_NUM_SECTORS=937703000 in tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml

Prepared internal

sudo BOARDID=3767 BOARDSKU=0005 ./tools/kernel_flash/l4t_initrd_flash.sh --network usb0 --no-flash -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" jetson-orin-nano-devkit internal

Prepared external

sudo BOARDID=3767 BOARDSKU=0005 ROOTFS_ENC=1 ./tools/kernel_flash/l4t_initrd_flash.sh --append --network usb0 --no-flash --showlogs --external-only --external-device nvme0n1p1 -S 64GiB -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_enc.xml -i ./sym2_t234.key ``-p "--generic-passphrase"`` jetson-orin-nano-devkit external

Flashed

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --network usb0 --flash-only

Failure of new type

Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for device to expose ssh ......RTNETLINK answers: File exists
RTNETLINK answers: File exists
Waiting for device to expose ssh ...Run command: flash on fc00:1:1:0::2
SSH ready
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /mnt/internal/flash.idx
Number of lines is 58
max_index=57
[ 1]: l4t_flash_from_kernel: Successfully create gpt for emmc
[ 1]: l4t_flash_from_kernel: Starting to create gpt for external device
Active index file is /mnt/external/flash.idx
Number of lines is 76
max_index=75
writing item=59, 6:0:primary_gpt, 512, 19968, gpt_primary_6_0.bin, 16896, fixed-<reserved>-0, 27fd0d0bebfc926e28e20774b22e1375a9a9303c
Error: Could not stat device /dev/mmcblk0 - No such file or directory.
Flash failure
Cleaning up...

Host logs:
flash_5-1_0_20240708-105825.log (7.0 KB)

Uart logs:
uart_01_preparation.log (105.7 KB)
uart_02_flashing.log (79 KB)

Suspicious part in flashing logs

NOTICE:  BL31: v2.6(release):346877e39
NOTICE:  BL31: Built : 12:32:40, Aug  1 2023
I/TC: Physical secure memory base 0x27c040000 size 0x3fc0000
I/TC: 
I/TC: Non-secure external DT found
I/TC: OP-TEE version: 3.21 (gcc version 9.3.0 (Buildroot 2020.08)) #2 Tue Aug  1 19:39:55 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: Test OEM keys are being used. This is insecure for shipping products!
E/TC:00 00 ekb_extraction_process:319 Tried all EKB_RKs but still can't extract the EKB image.
E/TC:00 00 jetson_user_key_pta_init:898 jetson_user_key_pta_init: Failed (ffff000f).
E/TC:00 00 call_initcalls:43 Initcall __text_start + 0x000f2848 failed
I/TC: Primary CPU switching to normal world boot
яб
Jetson UEFI firmware (version 4.1-33958178 built on 2023-08-01T19:34:02+00:00)
18

please inital a new discussion thread for follow-up.
it’s bad practice to reply your own thread with new questions.

1 Like
19

Just created

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.