Failure to unlock 'crypt_root' with fused module when ROOTFS_ENC=1

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
, ,
1

Configuration

  • Stock JetPack 5.1.2 / L4T r35.4.1 with this NVIDIA-vetted hotfix
  • Two NVIDIA Xavier AGX DevKits (conf="jetson-agx-xavier-devkit")
    • One is stock and unfused, all fuses are as per factory defaults
    • One is fused into production mode with SBK, PKC, and KEK0/1/2

Observations

Both fused and unfused devices can be flashed to internal storage and booted successfully using both the traditional flash.sh and l4t_initrd_flash.sh.

Both fused and unfused devices can be flashed to internal storage using eitherflash.sh orl4t_initrd_flash.sh when ROOTFS_AB=1 and ROOTFS_ENC=1.

However, neither fused or unfused devices can boot after flashing, if ROOTFS_AB=1 and ROOTFS_ENC=1.

From the console boot log, both fused and unfused give the same error:

ERROR: fail to unlock the encrypted dev /dev/mmcblk0p3.

Setup Details

We have followed the L4T r35.4.1 documentation closely and have no customizations.

Our flashing setup shell scripts use the following commands:

boot_keys_args=(
    -u "${boot_keys}/pkc.private" # for unfused modules, comment this line out
    -v "${boot_keys}/sbk.key"     # for unfused modules, comment this line out
)

echo "${ekb_key}" > "ekb.key"

conf="jetson-agx-xavier-devkit"

ROOTFS_AB=1 ROOTFS_ENC=1 \
    sudo --preserve-env=ROOTFS_AB,ROOTFS_ENC \
        bash -x \
            flash.sh \
                -i "ekb.key" \
                "${boot_keys_args[@]}" \
                --uefi-keys "uefi_keys/uefi_keys.conf" \
                "${conf}" internal

I emphasize that the only difference in the flashing setup is presence or absence of boot_keys_args in the above.

Further Investigation

To help track this down, we rebuilt l4t_initrd.img to introduce additional logging. Note that no functionality was changed, only extra logging was added.

Note that all keys in any logging output herein are insecure and are safe for public disclosure.

From the log files:

fused initrd log:

nvluks-srv-app -u -c '6f1f2487-2a44-4682-adaa-f06ab112d966' -> 'TEEC_InvokeCommand failed 0xffff0001 origin 0x4'

unfused initrd log:

nvluks-srv-app -u -c '9c448cf0-32a7-4283-85e6-628a94af083b' -> 'd4024a369b1a0dcf92f8a981f1053885'

So d4024a... is the LUKS password for filesystem UUID 9c448cf0...

However:

tools/disk_encryption/gen_luks_passphrase.py -k ekb.key -u -e 0x8[redacted]0 -c 9c448cf0-32a7-4283-85e6-628a94af083b -> d3dbc7bfd1f960e57e5b94ba3db6adde

… which is … a different password, and incorrect: d4024a... is incorrect, d3dbc7... is correct. This has been verified by manually luks-mounting system_root_encrypted.img.raw.

The only difference is fused vs unfused - everything else is 100% identical.

Neither can unlock the luks partition, but for different reasons!

2

hello andrew.fernandes,

let’s have quick confirmation for your EKS image,
please check those 4 magic bytes of your customize EKS image,
you may see-also Topic 271974.

3

So I just verified that all EKS images have the correct 00000024 45 45 4b 42 |EEKB| signature.

After a lot of review, I think the issue is that the documentation is … perhaps a bit misleading with how the encryption works with the fused keys. We have been very careful to follow the L4T developer documentation very closely, so I need to run a couple of tests to check my suspicions.

I’m going to run a couple of tests and update this thread over the next few days.

4

hello andrew.fernandes,

it would be great if you will share those for reference, thanks!

5

Well, on an un-fused Xavier AGX module, it seems you do not use the

... -i "./ekb.key" ...

argument as detailed here…

(Remember that we have both ROOTFS_AB=1 and ROOTFS_ENC=1…!)

So now we can get the un-fused module booting except for the following issue:

Problem With UEFI Keys

We receive the following notice on the console during boot for the un-fused module, even with custom UEFI keys:

Jetson UEFI firmware (version 4.1-33958178 built on 2023-08-01T19:34:02+00:00)
ESC   to enter Setup.
F11   to enter Boot Manager Menu.
Enter to continue boot.
**  WARNING: Test Key is used.  **
......

UEFI Key Configuration:

cd "${l4t}" # Linux_for_Tegra

cp -rv "${uefi_keys}" "./uefi_keys"
chmod u+w "./uefi_keys" "./uefi_keys"/*

tools/gen_uefi_default_keys_dts.sh "uefi_keys/uefi_keys.conf"
chmod 0640 uefi_keys/_out/*.auth

Flasher creation:

bash -x \
    tools/kernel_flash/l4t_initrd_flash.sh \
        "${boot_keys_args[@]}" \
        --uefi-keys "uefi_keys/uefi_keys.conf" \
        --no-flash --massflash "1" \
        "${conf}" internal

We will be trying a fuse-burned Xavier AGX next.

6

hello andrew.fernandes,

to clarify…
you don’t need to fuse the target (i.e. PKC, SBK) to enable disk encryption.
LUKS disk encryption support with a specific key. you should execute the script file, gen_ekb.py to generate an image. also, in the developer guide, [Tool for EKB Generation] that sym2.key is equivalent to ekb.key
please see-also Topic 270934, we’ve also check disk encryption with a custom key, it has worked normally with non-fused target.

as for WARNING: Test Key is used. message,
it’s reported by UEFI. you may refer to the public sources,
for example, edk2/MdeModulePkg/Universal/BdsDxe/BdsEntry.c at master · tianocore/edk2 · GitHub
this is normal message since the [UEFI Secureboot] has not enabled.

7

Hello, @JerryChang!

After much testing, we think we have secure + encrypted boot working, but are seeing some rather strange behaviour that we’d appreciate information on.

Setup

AGX Xavier (chip=t194), l4t-35.4.1, both ROOTFS_AB=1 and ROOTFS_ENC=1.

Fuses are burned, and the module is fused into production mode.

  • KEK0, KEK1, KEK2 are set and nonzero, with KEK256 = KEK0 + KEK1
  • Both the PKC and SBK are fused to nonzero keys

UEFI Keys

uefi_keys/
├── db_1.crt
├── db_1.key
├── db_2.crt
├── db_2.key
├── KEK.crt
├── KEK.key
├── PK.crt
├── PK.key
├── uefi_keys.conf
└── user.key

Note the presence of a nonzero user.key that will also be used as the sym_${chip}.key below.

Other Keys

    echo "00000000000000000000000000000000" > "${work}/sym2_${chip}.key"
    echo "bad66eb4484983684b992fe54a648bb8" > "${work}/fv_ekb_${chip}" # compiled-in OP-TEE default

That sym2_${chip}.key must be all-zero or root device decryption fails and the kernel panics.

EKS Generation

With the above setup, using the OP-TEE l4t-35.4.1 sources and samples:

    "${work}/venv/bin/python" \
        "${repo}/tools/gen_ekb.py" \
            -chip "${chip}" \
            -kek2_key "${work}/kek2_${chip}.key" \
            -fv "${work}/fv_ekb_${chip}" \
            -in_sym_key "${work}/sym_${chip}.key" \
            -in_sym_key2 "${work}/sym2_${chip}.key" \
            -out "${l4t}/bootloader/eks_${chip}.img"

Standalone Flasher Generation

    tools/kernel_flash/l4t_initrd_flash.sh \
        "${boot_keys_args[@]}" \                  # <- Sets the PKC and SBK Files
        --uefi-enc "uefi_keys/user.key" \         # <- identical to sym_${chip}.key
        --uefi-keys "uefi_keys/uefi_keys.conf" \
        --no-flash --massflash "1" \
        "jetson-agx-xavier-devkit" \
        "internal"

Results

After flashing, the boot succeeds, and we see UEFI Secure Boot is enabled.

However, we see the following oddities on the console log:

Jetson UEFI firmware (version 4.1-33958178 built on 2023-08-01T19:34:02+00:00)
ESC   to enter Setup.
F11   to enter Boot Manager Menu.
Enter to continue boot.
**  WARNING: Test Key is used.  **
......
I/TC: Reserved shared memory is disabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
L4TLauncher: Attempting Direct Boot
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0006
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
OpenAndReadFileToBuffer: \boot\initrd failed signature verification: Security Violation
ExtLinuxBoot:sds Failed to Authenticate \boot\initrd (Security Violation)
L4TLauncher: Unable to boot via extlinux: Security Violation
L4TLauncher: Attempting Kernel Boot
EFI stub: Booting Linux Kernel...
EFI stub: UEFI Secure Boot is enabled.
EFI stub: Using DTB from configuration table
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Exiting boot services and installing virtual address map...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
I/TC: Secondary CPU 2 initializing
I/TC: Secondary CPU 2 switching to normal world boot
I/TC: Secondary CPU 3 initializing
I/TC: Secondary CPU 3 switching to normal world boot
I/TC: Secondary CPU 4 initializing
I/TC: Secondary CPU 4 switching to normal world boot
I/TC: Secondary CPU 5 initializing
I/TC: Secondary CPU 5 switching to normal world boot
I/TC: Secondary CPU 6 initializing
I/TC: Secondary CPU 6 switching to normal world boot
I/TC: Secondary CPU 7 initializing
I/TC: Secondary CPU 7 switching to normal world boot
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x4e0f0040]
[    0.000000] Linux version 5.10.120-tegra (buildbrain@mobile-u64-6422-d7000) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2020.08) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP PREEMPT Tue Aug 1 12:32:50 PDT 2023
[    0.000000] Machine model: Jetson-AGX

Questions

  • Why are we still seeing WARNING: Test Key is used?

    • There are no test keys on the system at this point.
    • If we use a nonzero sym2.key, the system fails to unlock the root filesystem, fails to boot, and still gives the test key warning!

  • Why are we seeing TEE_InvokeTACommand failing?

    • Is this due to the (required) zero sym2.key that is not, as far as we can tell, used for anything?

  • Why are we seeing Security Violation for both \boot\initrd and extlinux?

    • Is extlinux simply not used when ROOTFS_ENC=1?
    • Are these “spurious failures”, meaning “Correctly failing as designed?”
    • This would imply that extlinux is not usable with UEFI and secure boot, correct?

Thank you!

8

hello andrew.fernandes,

there’re two separate issues, (1) UEFI secureboot, (2) ROOTFS_ENC.
I suggest you file another ticket for tracking UEFI secureboot.
let me have some reply below.

  1. UEFI.
    it’s due to below code snippets. had you create the db/dbx keys, and update the UEFI payloads?
    anyways, let’s use another new forum thread for tracking.
  //
  // If any component set PcdTestKeyUsed to TRUE because use of a test key
  // was detected, then display a warning message on the debug log and the console
  //
  if (PcdGetBool (PcdTestKeyUsed)) {
    DEBUG ((DEBUG_ERROR, "**********************************\n"));
    DEBUG ((DEBUG_ERROR, "**  WARNING: Test Key is used.  **\n"));
    DEBUG ((DEBUG_ERROR, "**********************************\n"));
    Print (L"**  WARNING: Test Key is used.  **\n");
  }
  1. ROOTFS_ENC

as you can see in the op-tee sample, optee/samples/hwkey-agent/host/tool/gen_ekb/example.sh,
that’s using $ echo "00000000000000000000000000000000" > sym2_t194.key as default sym2 key.
you cannot use nonzero vales if you did not update the EKS image as well.
FYI, please see-also Topic 258603 for the steps to enable disk encryption on Xavier with non-zero keys.

9

Thanks, @JerryChang … I have created Topic 279304.

From the above, note that:

  • the “Test key is used” warning happens even if sym2.key is nonzero, and

  • we explicitly regenerate the EKS image every time using gen_ekb.py

So we should not be seeing the indicated warnings and errors.

10

let’s following up Topic 279304 for "Test key is used" warning messages.
had you see-also Topic 258603 for the steps to enable disk encryption on Xavier with non-zero keys?

11

Following up on Topic 279304.

Note that Topic 258603 is not relevant to us since we are using “per device” keys based on the ECID.

We have tried and cannot use the -i option to flash.sh as this results in root filesystems that cannot be unlocked.

Further note that if we use the zero sym2.key, we complete boot successfully and our root filesystem is correctly encrypted with the correct per-device KEK2-derived key. We have verified this manually.

12

hello andrew.fernandes,

please check below to verify EKS image has update correctly,

On Target
you may running nvluks-srv-app on the target.
for instance, $ sudo /usr/sbin/nvluks-srv-app -c "luks-srv-ecid" -u
Result… 728eda7817027e6370774131e823763f

On Host
please check whether it shows the same by running the python script on the host,
$public_sources/r35.4.1/atf_and_optee/optee/samples/luks-srv/host/tool/gen_luks_passphrase/gen_luks_passphrase.py
for example,

# BR_CID: 0xE1012344705E35DE2C0000000BFF01C0          //Revise as your BR_CID
$ echo "f0e0d0c0b0a001020304050607080900" > ekb.key   //Revise as your EKB key
$ python3 gen_luks_passphrase.py -k ekb.key -c "luks-srv-ecid" -u -e "BR_CID"

Result… 728eda7817027e6370774131e823763f

13

The EKS image was verified to be created correctly.

We modified the initrd to dump the keys that were reported by the OP-TEE TA, and for the flashing scripts to also dump the keys that were used to generate the encrypted disk images.

For the case(s) above where we could boot successfully (zero sym2.key), the LUKS passwords were identical and the root volume was unlocked.

If sym2.key was nonzero or if the -i option was used — we tested both — the LUKS passwords did not match.

14

hello andrew.fernandes,

please also check you’ve really burned the KEK2 fuse correctly.
for Xavier series, KEK2 is the root EKB key for decryption and encryption,

there’s script file, odmfuseread.sh which could dump the fuse values of the target for quick confirmation,
thanks

15

We can verify that KEK2 is correctly burned.

The module is fused into production mode, so KEK2 is unreadable. However, the readable fuses, such as the PKC hash, are correct.

Moreover, we have used this module extensively with trusty on JetPack 4 for KEK2 + FV based LUKS unlock.

Lastly, the device-specific ECID-based-plus-KEK2 + FV produces correct keys, and we can successfully unlock both the disk image and boot to full userland on the module.

16

hello andrew.fernandes,

note,
we’ve test this locally to enable disk encryption on Xavier with non-zero keys.

FYI,
for Xavier series, the kek2.key is used to encrypt EKS image, (we use the zero key for the unfused boards) the key for decrypting kernel/kernel-dtb is stored in the EKS, it is passed with "-in_sym_key sym_t194.key" in gen_ekb.py. We call it "user_key" in JP-4, and changes it to "--uefi-enc" in JP5.
the disk encryption key (ekb.key) is also stored in the EKS, it is passed with "-in_sym_key2 sym2_t194.key" in gen_ekb.py python script file.

here’re more details,
for JP-4, it’s using with the option "--user_key" (which is only supported in JP-4) to encrypt/decrypt user images,
for JP-5.1.2, (please don’t use "--user_key" options in the JP-5) we’ve moving to use "--uefi-enc" to implement this feature with the UEFI secure boot.

I still doubt this is an issue of updating EKS image.
had you update the EKS image manually?
for example, here’s path of default EKS image, $OUT/Linux_for_Tegra/bootloader/eks_t194.img
you should overwrite that with your customized image which generated by gen_ekb python script.
also, you may add -r options to the flash.sh for re-flashing the target by reusing the existing binary. (in order to avoid system to overwrite those updated binaries again)

sample commands…
$ sudo ./flash.sh -r -u <pkc_keyfile> [-v <sbk_keyfile>] -i ekb.key --uefi-keys uefi_keys/uefi_keys.conf <target> mmcblk0p1

17

Hi, @JerryChang -

Turns out the problem is with the UEFI firmware as per Topic 279304.

We have carried out and verified all the steps above, as per JP5.

18

all right, thanks for confirmation. let’s moving to Topic 279304 for follwoing-up.

19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.