Jetson won't boot after update TOS image

Hi,

I’m testing the OP-TEE feature on JetPack 5.1. I’m following the instructions from atf_and_optee_README.txt but at the end it says:

Verifying the Image
----------------------------------------------------------------------
To verify the image:
1. Replace the default TOS image file with the newly generated TOS
   image. The default TOS image file is located at:
   <Linux_for_Tegra>/bootloader/tos-optee_t<platform>.img

2. Perform either of these tasks:
   - Flash the system as normal.
     This is useful for flashing a new system or replacing the
     entire operating system.
   - Re-flash the TOS image using these partition flash commands:
       sudo ./flash.sh -k secure-os <your_board_conf_file> mmcblk0p1
       ex:
       sudo ./flash.sh -k secure-os jetson-xavier-nx-devkit mmcblk0p1

3. Copy all the files under ./optee/install/t<platform> to the target.

When I generate the TOS.img, I used the command that comes along with the instructions:

Generate the tos.img with the commands:
   ./gen_tos_part_img.py \
       --monitor ./atf_build/arm-trusted-firmware/build/tegra/t<platform>/release/bl31.bin \
       --os ./optee/build/t<platform>/core/tee-raw.bin \
       --dtb ./optee/tegra<platform>-optee.dtb \
       --tostype optee \
       ./tos.img

So I change the name of the tos.img for tos-optee_t194.img and move it to the bootloader.
Then I reflash the entire OS. And it won’t boot. It get stuck in the booting process showing some logs like:

failed to start nvpmodel
failed to start refresh fwupd
see systemctl status fwupd-refresh.system

Am I missing one step of the instructions here? Or there is an extra step that I’m not taking?

Best,

JDiego Delgado
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com/
Website: www.ridgerun.com

hello jdiegodelgado,

it looks incorrect naming,
please refer to flash configuration file, i.e. TOSFILE="bootloader/tos_t194.img";
please update the TOS binary naming and please try again.
thanks

JerryChang,

I renamed the file from tos.img to tos_t194.img, then moved it to the bootloader directory, and finally re-flashed the whole OS again. But I got the same result. It won’t boot.
Am I missing something?

Thanks!

hello jdiegodelgado,

could you please share your target flashing messages and also serial logs for system boot-up.

JerryChang,

Thanks for your support. I enabled the serial communication by using minicom. Please find the logs attached to this post.

Strange thing is that I could configure the platform by using the serial port. Before that I was using the GUI option to try to configure the platform (select language, region, user name, password, etc, etc) but it got stuck in the boot logs every time.
Now it seems that I could to it but using the serial console.

minicom.cap (91.0 KB)

Now I have some doubts about OP-TEE. How can I check that the new TOS image was flashed? I added the “hello world” trust app from this link: GitHub - linaro-swg/optee_examples: OP-TEE Sample Applications to Linux_for_Tegra/source/public/nvidia-jetson-optee-source/optee/samples and modified the Makefiles to build it.
How can I know that this worked? How can I see this “hello world”?

Also, what these logs mean? Should I be worried for them?

[  262.288465] loop: disagrees about version of symbol module_layout
[  273.908896] zram: disagrees about version of symbol module_layout
[  275.073970] mtd: disagrees about version of symbol module_layout
[  275.379002] fuse: disagrees about version of symbol module_layout
[  597.928165] fuse: disagrees about version of symbol module_layout

Best,

JerryChang,

Another doubt, in the atf_and_optee_README.txt instructions, after f;ashing the board, there’s a step that says:

Copy all the files under ./optee/install/t<platform> to the target.

Which location could be a good place to put these files within the platform?

Best,

hello jdiegodelgado,

it looks it’s stuck here for waiting system configuration.

[   32.806104] Please complete system configuration setup on the serial port provided by Jetson's USB device mode connection. e.g. /dev/ttyUSBx where x can 0, 1, 2 etc.

you may refer to Skipping oem-config.
could you please try running below $OUT/Linux_for_Tegra/tools/l4t_create_default_user.sh to creates a default user accounts for testing.

JerryChang,

I tested skipping the oem-config by running the /Linux_for_Tegra/tools/l4t_create_default_user.sh. It seems to work fine (please see the logs attached below).

But some logs called my attention, for instance this one:

E/TC:0 0 jetson_user_key_pta_init:700 jetson_user_key_pta_init: Failed (f0100006).

Why does jetson_user_key_pta_init failed? How do I know that the new Trusted OS is working properly?

minicom.cap (83.6 KB)

Also, I noticed that inside the Linux_for_Tegra/bootloader/ path, originally you find two symlink files:

  • tos_t194.img
  • tos_t234.img

They look like this: tos_t234.img -> tos-optee_t234.img.
When I created my new tos.img, should I renamed it for tos_t194.img, move it to the Linux_for_Tegra/bootloader/ path, and then made it a symlink that points to tos-optee_t194.img?

Best,

Hi @JerryChang ,

Can you please provide guidance on this doubt?

Thanks!

hello jdiegodelgado,

yes, for running gen_tos_part_img.py, you should given different TOS image names for different platforms.
you may see-also flash configuration file.
Orin series, TOSFILE="bootloader/tos_t234.img";
Xavier series, TOSFILE="bootloader/tos_t194.img";

you may looking bootloader logs.
there’ll be BL31 and also OP-TEE version and built times. it shall be match your host machine build time after your TOS image has updated.

had you flash user_key which is specified in the eks.img.
there’s EKB (Encrypted Binary Blob) stores two keys, one is the kernel encryption key, and another one is the LUKS key for disk encryption support.
it’s boot-up process to derive keys from SE keyslots. this is error reported if there’s no keys available.

@JerryChang ,

Thanks again for your support. I was able to fixed the jetson_user_key_pta_init failed by generating the EKB key.
Now my question would be regarding the step 3 of the atf_and_optee_README.txt, precisely on the Generating the tos.img with ATF and OP-TEE images section, which says:

3. Copy all the files under ./optee/install/t<platform> to the target.
Now my next question is about the "helloworld ta" that I added to the OP-TEE sou

Where exactly should I copy these files within the Jetson Xavier AGX platform?

Best,

hello jdiegodelgado,

there are 3 folders in optee/install/t<platform>, i.e. /bin, /lib and /usr.
just copy those binaries to the corresponding folders, /bin, /lib and /usr in the target board.

Hi @JerryChang ,

Thanks for your response. On Friday I copied the binaries to the corresponding folders in the target board. But I started to experience some issues when I tried to run a trusted application I got some errors.

This is using the op-sources from JetPack 5.1:

When I run xtest I got:

Run test suite with level=0
TEE test application started over default TEE instance
Failed to open TEE context: 0xffff0008

Then, I tried to run the tee-supplicant:

$ tee-supplicant
ERR [2340] TEES:main:905: failed to find an OP-TEE supplicant device

Then, I checked if the tee-supplicant was running, but it seems that it was not:

$ ps | grep supplicant
NO OUTPUT

So I tried to run it as daemon, but I got:

$ tee-supplicant -d
ERR [2492] TEES:main:891: make_daemon(): -1

Next, I checked the logs to find if something went wrong during boot process and checked that the tee supplicant did start at boot. But I did not see any error:
minicom.cap (162.0 KB)

Confirm that tee-supplicant did start at boot:
[ 9.539517] systemd[1]: Started OP-TEE Client Supplicant.

After that, I checked if the tee driver was created, which it was:

$ ls /dev/tee*
/dev/tee0  /dev/teepriv0

Then, I checked if the driver was loaded:

$ sudo dmesg | grep optee
[    4.186253] optee: probing for conduit method.
[    4.187181] optee: revision 3.19 (30cb55b2)
[    4.188252] optee: dynamic shared memory is enabled
[    4.189558] optee: initialized driver

Do you have any suggestions why xtest is failing and tee-supplicant seems not be running?

Best,

BTW, for the user TAs (i.e. the TA users create), they should be put under… /lib/optee_armtz.

@JerryChang ,

Thanks for your answer. But the problem that I’m facing is with JetPack5.1 OP-TEE fresh sources. Yesterday I re-flashed the board again, just to test the OP-TEE that comes with JetPack5.1, and check if it works.
On my last post I was debugging the built-in OP-TEE. I did not add anything from my side.

Any thoughts on why is it failing?
I will tag @KevinFFF @WayneWWW to know if they have more insights about this.

$ tee-supplicant
ERR [2340] TEES:main:905: failed to find an OP-TEE supplicant device
$ xtest
Run test suite with level=0
TEE test application started over default TEE instance
Failed to open TEE context: 0xffff0008

Best,

hello jdiegodelgado,

that’s error reported by getting device fd.
could you please check whether you have sysnode like below… /dev/teepriv*

Hi @JerryChang ,

Yes, the devices seems to be correctly created:

$ ls /dev/tee*
/dev/tee0  /dev/teepriv0

Best,

Hi @JerryChang ,

Is there anything else that I can tried from my side?

This error is the one that is getting my attention:

$ tee-supplicant
ERR [2340] TEES:main:905: failed to find an OP-TEE supplicant device

Best,

hello jdiegodelgado,

we are able to reproduce the same issue locally, and we’ve created an internal thread for tracking.

since optee has booted up properly according to the logs.
did you run these commands in root permission BTW? please have a try with… $ sudo tee-supplicant