Secure boot with initrd flash failed

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi Guys, I am working on the secure boot on TX2 NX, and the background is we still use version of 32.7.4, running on NVME SSD.
we have full disk encryption on top of AB scheme, and it is all working fine, and now try to get the secure boot.
first I burned the fuse like below

sudo ./odmfuseread.sh -i 0x18 -k ./rsa_priv.pem -S ./SKB.txt jetson-xavier-nx-devkit-tx2-nx

then I generate the signed firmware etc using blow

ROOTFS_AB=1 NO_ROOTFS=1 ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash  -u ./rsa_priv.pem -v ./SKB.txt --showlogs jetson-xavier-nx-devkit-tx2-nx internal
ROOTFS_ENC=1 ROOTFS_AB=1  ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -S 40GiB  -u ./rsa_priv.pem -v ./SKB.txt --showlogs -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc_ab.xml --external-only --append jetson-xavier-nx-devkit-tx2-nx external

And these steps seems run OK.
the log is here
1.txt (77.1 KB)

then I program using below command:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only -u ./rsa_priv.pem -v ./SKB.txt

then it has the problem to wait unit boot up in initrd.

***************************************
*                                     *
*  Step 3: Start the flashing process *
*                                     *
***************************************
Waiting for target to boot-up...
Waiting for target to boot-up...
...
Waiting for target to boot-up...
Timeout
Cleaning up...

the error message seems related to some signature verification, not very sure

[0010.384] I> ########## Fixed storage boot ##########
[0010.389] I> Loading kernel from blob
[0010.392] I> Found imgtype:20 in blob @ idx:8, offset:3686728, size: 629136
[0010.399] I> Load address: 0x84b84148
[0010.403] I> Validate kernel ...
[0010.406] I> T18x: Authenticate kernel (bin_type 24), max size 0x4000000
[0010.415] I> Checking boot.img header magic ... [0010.419] E> Invalid header magic
[0010.422] E> Storage boot failed, err: 724238360
[0010.427] E> Error (724238360) builtin kernel/dtb load failed
[0010.432] I> Filling _next_stage_param: ep: 0x800040d9c, dtb: 0xffffffff
[0010.439] I> TBoot-CPU Recovery done

it appears the fuse has been program then initrd cannot run? is initrd require signed too? and how to debug the reason for that?
thanks

3

hello jiangpen,

let’s narrow down the issue,
please confirm you’re able to have image flashing to fused device, and boot up correctly.

4

hi @JerryChang , the issue is I am adding secure boot to full disk encryption.
The full disk encryption is working (boot up) without fused device.
Once I fused it, I cannot program the device anymore. I am using initrd to program and I think have to stick to this method.
The observation is once the device is fused, then flash initrd cannot run anymore.
The full disk encryption method we have is internal eMMC not encrypted, but external NVME encrypted.
thanks

5

hello jiangpen,

it should be able to re-flash once you’re assign correct PKC/SBK keys.
let’s give it a try with below anyways,
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash -u ./rsa_priv.pem -v ./SKB.txt --showlogs jetson-xavier-nx-devkit-tx2-nx internal
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -u ./rsa_priv.pem -v ./SKB.txt --showlogs -c ./tools/kernel_flash/flash_l4t_external.xml --external-only --append jetson-xavier-nx-devkit-tx2-nx external
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only -u ./rsa_priv.pem -v ./SKB.txt

6

hi @JerryChang , I tried again, the first two lines run fine, but the last one failed.
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only -u ./rsa_priv.pem -v ./SKB.txt
I think the failure is the flash initrd kernel download to device, but the initrd kernel signature verification failed.
the programming script exit as show

Waiting for target to boot-up...
Timeout
Cleaning up...

and the console for usart show signature verification failed. Device has been fused in this stage.

[0012.907] I> Recovery boot_type: 1
[0012.915] I> fixed regulator driver initialized
[0012.958] I> register 'maxim' power off handle
[0012.965] I> virtual i2c enabled
[0012.968] I> registered 'maxim,max77620' pmic
[0012.973] I> tegrabl_gpio_driver_register: register 'max77620-gpio' driver
[0012.979] I> Blob header @ 0x86200000
[0012.983] I> ########## Fixed storage boot ##########
[0012.988] I> Loading kernel from blob
[0012.991] I> Found imgtype:20 in blob @ idx:8, offset:3686728, size: 63390096
[0012.998] I> Load address: 0x86584148
[0013.002] I> Validate kernel ...
[0013.005] I> T18x: Authenticate kernel (bin_type 24), max size 0x4000000
[0013.254] E> SHA verification failed for header 1
[0013.259] E> Storage boot failed, err: 1075118360
[0013.263] E> Error (1075118360) builtin kernel/dtb load failed
[0013.269] I> Filling _next_stage_param: ep: 0x800040d9c, dtb: 0xffffffff
[0013.275] I> TBoot-CPU Recovery done

may I know the procedure of the burn fuse and programing, is burn fuse first or programming first?

9

hello jiangpen,

is it a must to program secureboot with initrd flash script?
please see-also Topic 285586, comment #34 for the steps we’ve fuse/flash TX2 NX.

10

Hi @JerryChang , my user case is system boot to NVME and NVME has a unencrypted app partition which set up and jump to the encrypted partition. So I need program the external NVME which requires the initrd.
may I know if possible to flash with initrd with a fused device? if yes, any guideline please.
thanks
also could you please explain a little more about the initrd boot up procedure to me?
I think what is happening is a kernel in initrd which failed to pass verification , may I know how this initrd kernel etc has been signed etc?

also I checked the README_initrd_flash.txt, and it appears it can support initrd program from a fused device, but I just cannot get it working. Any help are appreciated.

11

hello jiangpen,

did you refer to [Workflow 5: Secureboot] for the steps?

12

yes, I did,
I program fuse using below command

sudo ./odmfuse.sh  -i 0x18 -k ./rsa_priv.pem -S ./SKB.txt   jetson-xavier-nx-devkit-tx2-nx

then I use similar way like workflow 5 (I need a little change as add '–external-only --append ’ )

sudo ROOTFS_ENC=1 ROOTFS_AB=1  ./tools/kernel_flash/l4t_initrd_flash.sh  --external-device nvme0n1p1 -S 40GiB  -u rsa_priv.pem -v SKB.txt  --showlogs -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_enc_ab.xml --external-only --append jetson-xavier-nx-devkit-tx2-nx external

but the device shows the signature verification failed in initrd

[0015.327] I> Recovery boot_type: 1
[0015.335] I> fixed regulator driver initialized
[0015.378] I> register 'maxim' power off handle
[0015.385] I> virtual i2c enabled
[0015.388] I> registered 'maxim,max77620' pmic
[0015.392] I> tegrabl_gpio_driver_register: register 'max77620-gpio' driver
[0015.399] I> Blob header @ 0x86200000
[0015.403] I> ########## Fixed storage boot ##########
[0015.408] I> Loading kernel from blob
[0015.411] I> Found imgtype:20 in blob @ idx:8, offset:3686728, size: 63390096
[0015.418] I> Load address: 0x86584148
[0015.421] I> Validate kernel ...
[0015.424] I> T18x: Authenticate kernel (bin_type 24), max size 0x4000000
[0015.674] E> SHA verification failed for header 1
[0015.679] E> Storage boot failed, err: 1075118360
[0015.683] E> Error (1075118360) builtin kernel/dtb load failed
[0015.689] I> Filling _next_stage_param: ep: 0x800040d9c, dtb: 0xffffffff

I tried to use flash.sh, it can manage to flash internal eMMC for fused device, but cannot do NVME, so I have to use the initrd way.
any suggestion are welcome.

I also verified that if using same program command on an unfused unit, it even cannot connect to generate image, which indicates the PKC key actually works in fused unit. Fused unit can connect device to generate image etc, but cannot program with initrd in the last step.
thanks

13

just curious where did you obtain such xml file, or, it’s your customize xml file?

14

yes, I created this file to support the full disk encryption and A/B scheme. This part is fully working.
Now I need add secure boot on top of that.
it is really appreciate if you can have a look of my questions and give some advice on these.
thanks

15

hello jiangpen,

could you please attach your l4t_sign_image.sh and tegraflash_internal.py for cross-check.
besides, is it a must to stay-on r32.7.4? is it possible to verify on the latest rel-32 public release version?

16

tegraflash_internal.py.txt (160.2 KB)
l4t_sign_image.sh.txt (8.3 KB)
here they are, yes, it seems r32.7.4 is latest one to support TX2 NX
thanks @JerryChang

update, today I make a little progress, I found the boot0.img which is not signed by default, so I signed it with the key, I found the the error log is different now

[0013.632] I> Validate kernel ...                                                                                                                        
[0013.635] I> T18x: Authenticate kernel (bin_type 24), max size 0x4000000                                                                                
[0013.884] I> Checking boot.img header magic ... 
[0013.889] E> Invalid header magic                                                                      
[0013.892] E> Storage boot failed, err: 724238360                                                                                                        
[0013.896] E> Error (724238360) builtin kernel/dtb load failed                                                                                           
[0013.902] I> Filling _next_stage_param: ep: 0x800040d9c, dtb: 0xffffffff
//old one below is SHA for header1, now it is boot.img invalid header magic
[0015.421] I> Validate kernel ...
[0015.424] I> T18x: Authenticate kernel (bin_type 24), max size 0x4000000
[0015.674] E> SHA verification failed for header 1
[0015.679] E> Storage boot failed, err: 1075118360
[0015.683] E> Error (1075118360) builtin kernel/dtb load failed
17

hello jiangpen,

let’s narrow down the issue, are you able to flash the board correctly through flash script?
please see-also How to Flash an Encrypted Rootfs to an External Storage Device to flash an encrypted rootfs.

18

Hi @JerryChang
with a none-fused board, I can flash board correctly with encrypted rootfs to NVME and bootup successfully. That part works.
Now I fused the board, and add PKC and SBK keys as parameter to flash board in same way (initrd), and now it failed to flash. It stuck in initrd.
thanks

19

hello jiangpen,

it’s flash script to flash internal storage (i.e. eMMC), and initrd for external storage (i.e. NVMe).

20

we use initrd to flash both internal and external , the procedure is as below. working fine.

ROOTFS_AB=1 NO_ROOTFS=1 ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash  --showlogs jetson-xavier-nx-devkit-tx2-nx internal
ROOTFS_ENC=1 ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -S 40GiB --showlogs -c $PARTITION_XML --external-only --append jetson-xavier-nx-devkit-tx2-nx external
./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --showlogs

Now I fused the board, and add -u for PKC and -v for SKB to above commands, then it does not work, stuck in initrd.

21

hello jiangpen,

please share the complete UART logs for checking.

22

@JerryChang please see uart log here
usart.txt (14.5 KB)
programming log is here
log_internal.txt (230.6 KB)
thanks

23

hello jiangpen,

let’s check you’ve fuse the target correctly.
please read the fuse to ensure all the fuse variable has programmed correctly, you may double check with/without keys.