UEFI Reports "Test Key Detected" even when there are no test keys on the system

This topic is a continuation of Topic 276872.

(Ping @JerryChang!)


Setup

AGX Xavier (chip=t194), l4t-35.4.1, both ROOTFS_AB=1 and ROOTFS_ENC=1.

Fuses are burned, and the module is fused into production mode.

  • KEK0, KEK1, KEK2 are set and nonzero, with KEK256 = KEK0 + KEK1
  • Both the PKC and SBK are fused to nonzero keys

UEFI Keys

uefi_keys/
├── db_1.crt
├── db_1.key
├── db_2.crt
├── db_2.key
├── KEK.crt
├── KEK.key
├── PK.crt
├── PK.key
├── uefi_keys.conf
└── user.key

Note the presence of a nonzero user.key that will also be used as the sym_${chip}.key below. (In other words, the user.key and sym_${chip}.key are identical.)

Other Keys

    echo "bad66eb4484983684b992fe54a648bb8" > "${work}/fv_ekb_${chip}" # compiled-in OP-TEE default

Below, we discuss two scenarios, one with a zero sym2.key and one with a nonzero sym2.key:

    # NOTE ONLY ONE LINE OF THE FOLLOWING ARE USED AT ANY ONE TIME!!!
    echo "00000000000000000000000000000000" > "${work}/sym2_${chip}.key"
    echo "206684d1a157bcdd3afec04925b113e3" > "${work}/sym2_${chip}.key"

Note that sym2_${chip}.key must be all-zero or root device decryption fails and the kernel panics. However this topic is about a strange UEFI notice that happens before the kernel is loaded.


EKS Generation

With the above setup, using the OP-TEE l4t-35.4.1 sources and samples:

    "${work}/venv/bin/python" \
        "${repo}/tools/gen_ekb.py" \
            -chip "${chip}" \
            -kek2_key "${work}/kek2_${chip}.key" \
            -fv "${work}/fv_ekb_${chip}" \
            -in_sym_key "${work}/sym_${chip}.key" \
            -in_sym_key2 "${work}/sym2_${chip}.key" \
            -out "${l4t}/bootloader/eks_${chip}.img"

Standalone Flasher Generation

    tools/kernel_flash/l4t_initrd_flash.sh \
        "${boot_keys_args[@]}" \                  # <- Sets the PKC and SBK Files
        --uefi-enc "uefi_keys/user.key" \         # <- identical to sym_${chip}.key
        --uefi-keys "uefi_keys/uefi_keys.conf" \
        --no-flash --massflash "1" \
        "jetson-agx-xavier-devkit" \
        "internal"

Results

We flash in two almost-identical ways:

  • one with sym2.key all zero, and
  • one with sym2.key nonzero.

After flashing, both boots succeed in launching the kernel.

Both boots succeed, with both showing UEFI Secure Boot is enabled.

The only difference is:

  • the zero sym2.key scenario unlocks the root filesystem and completes boot normally

  • the nonzero sym2.key scenario still has UEFI reporting a test key, correctly begins booting the kernel, but the root filesystem cannot be unlocked.

Logs for the “mass-flash” make, flash, and console boot:

Zero sym2.key:

Nonzero sym2.key:


In addition, we see the following oddities on the console log:

Jetson UEFI firmware (version 4.1-33958178 built on 2023-08-01T19:34:02+00:00)
ESC   to enter Setup.
F11   to enter Boot Manager Menu.
Enter to continue boot.
**  WARNING: Test Key is used.  **
......
I/TC: Reserved shared memory is disabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
L4TLauncher: Attempting Direct Boot
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0006
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
E/TA:  decrypt_image:99 TEE_InvokeTACommand failed with res = 0xffff0007
OpenAndReadFileToBuffer: \boot\initrd failed signature verification: Security Violation
ExtLinuxBoot:sds Failed to Authenticate \boot\initrd (Security Violation)
L4TLauncher: Unable to boot via extlinux: Security Violation
L4TLauncher: Attempting Kernel Boot
EFI stub: Booting Linux Kernel...
EFI stub: UEFI Secure Boot is enabled.
EFI stub: Using DTB from configuration table
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Exiting boot services and installing virtual address map...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
I/TC: Secondary CPU 2 initializing
I/TC: Secondary CPU 2 switching to normal world boot
I/TC: Secondary CPU 3 initializing
I/TC: Secondary CPU 3 switching to normal world boot
I/TC: Secondary CPU 4 initializing
I/TC: Secondary CPU 4 switching to normal world boot
I/TC: Secondary CPU 5 initializing
I/TC: Secondary CPU 5 switching to normal world boot
I/TC: Secondary CPU 6 initializing
I/TC: Secondary CPU 6 switching to normal world boot
I/TC: Secondary CPU 7 initializing
I/TC: Secondary CPU 7 switching to normal world boot
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x4e0f0040]
[    0.000000] Linux version 5.10.120-tegra (buildbrain@mobile-u64-6422-d7000) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2020.08) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #1 SMP PREEMPT Tue Aug 1 12:32:50 PDT 2023
[    0.000000] Machine model: Jetson-AGX

Questions

  • Why are we still seeing WARNING: Test Key is used?

    • There are no test keys on the system at this point.

    • If we use a nonzero sym2.key:

      • UEFI still gives the test key warning

      • the system fails to unlock the root filesystem

  • Why are we seeing TEE_InvokeTACommand failing?

    • Is this due to the (required) zero sym2.key?

    • As far as we can tell, sym2.key is unused, correct?

  • Why are we seeing Security Violation for both \boot\initrd and extlinux?

    • Is extlinux simply not used when ROOTFS_ENC=1?

    • Are these “spurious failures”, meaning “Correctly failing as designed?”

      • This would imply that extlinux is not usable with UEFI and secure boot, correct?

Thank you!

Hi andrew.fernandes,

Are you using AGX Xavier or Xavier NX?
Is it on the devkit or custom board?

Could you add debug message in DetectTestKey.c to check if PcdTestKeyUsed is enabled by the following line: edk2/FmpDevicePkg/FmpDxe/DetectTestKey.c at main-edk2-stable202308 · NVIDIA/edk2 · GitHub

Are you using AGX Xavier or Xavier NX?

AGX Xavier

Is it on the devkit or custom board?

Stock NVIDIA devkit

Could you add debug message in DetectTestKey.c

Will do, later today.

I’ve moved your topic to the correct category for AGX Xavier.

Thanks, it may be caused from the match with the test key. Please check how it triggers TestKeyUsed to be True.

Alright, I added debugging output, and here it is from the console log:

ripgrep --text 'FmpDxe\|' console.log
2920:FmpDxe|DetectTestKey: =*=> ENTRY <=*=
2921:FmpDxe|DetectTestKey: PublicKeyDataXdr => 0x85801C585 => [ 308203EC308202D4A003020102020900C091C5E2B766C0F8300D06092A864886F70D01010B0500308182310B300906035504061302434E310B300906035504080C025348310B300906035504070C02534831123010060355040A0C095469616E6F436F7265310E300C060355040B0C0545444B49493111300F06035504030C0854657374526F6F743122302006092A864886F70D010901161365646B6969407469616E6F636F72652E6F7267301E170D3137303431303038323734305A170D3137303531303038323734305A308182310B300906035504061302434E310B300906035504080C025348310B300906035504070C02534831123010060355040A0C095469616E6F436F7265310E300C060355040B0C0545444B49493111300F06035504030C0854657374526F6F743122302006092A864886F70D010901161365646B6969407469616E6F636F72652E6F726730820122300D06092A864886F70D01010105000382010F003082010A0282010100B929296C600CD723F67DEEF062FFD9C9AA558C8195563FB75653B0C28212C53B7523B94DD6C45573F3AA95A81BF3937E9E40E41D229C93070BD7AA5BD7E41A2184D7635903501FF5145593919BF552B0BF0E5C683B5952989656E1ABC443BB05577845019F5815530E11942F0EF1A619A26E86392B338DC7C5EBEE1E33D33294C159C40C970B12485F33F660747D57C2132D7DA987A335EA91833F677A921F01539F625F9912FD731B2D9E2B6C3449AF4F078FC0E96B9E5F7935DA2A5C88EEF64861DA96E34846A0941C9DF65C870EEF7409910D3D5AE7C54C8A7AACA185B667441755523AE8114D58A2930062EA7B80EDCFBDDF75804BB96563AD0B4D74FA590203010001A3633061301D0603551D0E0416041416AAD68E1B2D43F32DB024AD36653FB2FAB12CED301F0603551D2304183016801416AAD68E1B2D43F32DB024AD36653FB2FAB12CED300F0603551D130101FF040530030101FF300E0603551D0F0101FF040403020186300D06092A864886F70D01010B0500038201010095DEDFA414DB9222781ABD319D1ED72F0A10115D7461E830C4F315E93054F4BB0C0478135D2CDD8C9290D19CD0D018A3A3FC8C285AD4914D08C3F61AC8DDA60858E21595FB2D2D8AB13080BD9AB6E12C203EDDC4C75565CF2817F4EEDABE7770D552D6157AFBADAFFDD545905AE63142D784B349566AD347F3BF68608B0FE2AFF4E3EC12B9E23A16114E4D7379AF47854C76269E8B32C08EC2DC27A6EFAC939EA15ECF3445E02AC79D4DD7D7377297F858F9B63548F1D10A727FFD4D7CE9CCD8481B495253DE51015335BC90CD8C8ACC4320A745FF2B55B08B2DFF55154B84D0C3D3909C944B55D562EA22AB6268DD53C6DCA5DD9A2D8E797C2E9CE466808C1D ]
2922:FmpDxe|DetectTestKey: Digest => [ 2E97891BDBE708AA8CB28FAD20A983C7847D4FEE4825E94D39FA349AB8B1C426 ]
2923:FmpDxe|DetectTestKey: CompareMem (Digest, PcdGetPtr (PcdFmpDeviceTestKeySha256Digest), SHA256_DIGEST_SIZE) == 0
2924:FmpDxe|(NVIDIA System Firmware): Test key detected in PcdFmpDevicePkcs7CertBufferXdr.
2925:FmpDxe|DetectTestKey: =*=> EXIT <=*= TestKeyUsed: 1

This is the test key defined in nvidia-uefi/edk2/FmpDevicePkg/FmpDevicePkg.dec

  80   │ [PcdsFixedAtBuild]
  81   │   ## The SHA-256 hash of a PKCS7 test key that is used to detect if a test key
  82   │   #  is being used to authenticate capsules.  Test key detection is disabled by
  83   │   #  setting the value to {0}.
  84   │   # @Prompt SHA-256 hash of PKCS7 test key.
  85   │   gFmpDevicePkgTokenSpaceGuid.PcdFmpDeviceTestKeySha256Digest|{0x2E, 0x97, 0x89, 0x1B, 0xDB, 0xE7, 0x08, 0xAA,  0x8C, 0xB2, 0x8F, 0xAD, 0x20, 0xA9, 0x83, 0xC7,  0x84, 0x7D, 0x4F, 0xEE, 0x48, 0x25, 0xE9, 0x4D,  0x39, 0xFA,
       │  0x34, 0x9A, 0xB8, 0xB1, 0xC4, 0x26}|VOID*|0x40000009

Attached are the logs to:

  • make the standalone flasher
  • the flashing log itself, and
  • the console log

console.log (250.9 KB)
flash.log (96.8 KB)
make.log (355.3 KB)


Observation and Question

To our knowledge, we are not using any test keys in our setup. Attached are the uefi_keys from our Linux_for_Tegra directory, and you can verify that these are being used from the log above.

uefi_keys.zip (33.6 KB)

Note that we build the offline flasher using:

tools/kernel_flash/l4t_initrd_flash.sh \
        "${boot_keys_args[@]}" \                  # <- Sets the PKC and SBK Files
        --uefi-enc "uefi_keys/user.key" \         # <- identical to sym_${chip}.key
        --uefi-keys "uefi_keys/uefi_keys.conf" \
        --no-flash --massflash "1" \
        "jetson-agx-xavier-devkit" \
        "internal"

Where is the test key being used?

Our suspicion is that this has to do with the fact that we’ve set both ROOTFS_AB=1 and ROOTFS_ENC=1, but that is just a guess.

Well, we have repeated our tests with:

  • fused modules
  • unfused modules
  • all combinations of ROOTFS_AB and ROOTFS_ENC being enabled and disabled
  • standalone mfi_*, initrd-based flashing and normal old-style flash.sh flashing

With the same results: WARNING: Test Key is used.

We have also verified that the flashing scripts are including UefiDefaultSecurityKeys.dtbo, and that this dtbo contains our correct UEFI keys from the above. We also see that UefiDefaultSecurityKeys.dtbo is used as an argument to tegraflash.py.


Grasping at straws here, but we did notice that the tegraflash.py command had two commas before the UefiDefaultSecurityKeys.dtbo overlay list… it’s small thing, but could that be part of the problem?

./tegraflash.py --bl nvtboot_recovery_cpu_t194.bin --sdram_config tegra194-mb1-bct-memcfg-p2888.cfg,tegra194-memcfg-sw-override.cfg  --odmdata 0x9190000  --overlay_dtb L4TConfiguration.dtbo,tegra194-p2822-camera-dual-imx274-overlay.dtbo,tegra194-p2822-camera-e3326-overlay.dtbo,tegra194-p2822-camera-e3331-overlay.dtbo,tegra194-p2822-camera-e3333-overlay.dtbo,tegra194-p2822-camera-imx185-overlay.dtbo,tegra194-p2822-camera-imx390-overlay.dtbo,tegra194-p2888-0005-overlay.dtbo,tegra194-p2888-0001-p2822-0000-overlay.dtbo,,UefiDefaultSecurityKeys.dtbo  --bldtb tegra194-p2888-0001-p2822-0000.dtb --applet mb1_t194_prod.bin --cmd "flash; reboot" --soft_fuses tegra194-mb1-soft-fuses-l4t.cfg  --cfg flash.xml --chip 0x19 --uphy_config tegra194-mb1-uphy-lane-p2888-0000-p2822-0000.cfg --minratchet_config tegra194-mb1-bct-ratchet-p2888-0000-p2822-0000.cfg --device_config tegra19x-mb1-bct-device-sdmmc.cfg --misc_cold_boot_config tegra194-mb1-bct-misc-l4t.cfg --misc_config tegra194-mb1-bct-misc-flash.cfg --pinmux_config tegra19x-mb1-pinmux-p2888-0000-a04-p2822-0000-b01.cfg --gpioint_config tegra194-mb1-bct-gpioint-p2888-0000-p2822-0000.cfg --pmic_config tegra194-mb1-bct-pmic-p2888-0001-a04-E-0-p2822-0000.cfg --pmc_config tegra19x-mb1-padvoltage-p2888-0000-a00-p2822-0000-a00.cfg --prod_config tegra19x-mb1-prod-p2888-0000-p2822-0000.cfg --scr_config tegra194-mb1-bct-scr-cbb-mini.cfg --scr_cold_boot_config tegra194-mb1-bct-scr-cbb-mini.cfg --br_cmd_config tegra194-mb1-bct-reset-p2888-0000-p2822-0000.cfg --dev_params tegra194-br-bct-sdmmc.cfg,tegra194-br-bct_b-sdmmc.cfg  --bin "mb2_bootloader nvtboot_recovery_t194.bin; mts_preboot preboot_c10_prod_cr.bin; mts_mce mce_c10_prod_cr.bin; mts_proper mts_c10_prod_cr.bin; bpmp_fw bpmp-2_t194.bin; bpmp_fw_dtb tegra194-a02-bpmp-p2888-a04_lz4.dtb; spe_fw spe_t194.bin; tos tos-optee_t194.img; eks eks_t194.img; bootloader_dtb tegra194-p2888-0001-p2822-0000.dtb"   --secondary_gpt_backup  --bct_backup  --boot_chain A 

I could reproduce this issue locally on the devkit.
I’m checking this with internal and will update to you once there’s any result.

Could you apply the following patch to disable compiling the test key?

diff --git a/Platform/NVIDIA/NVIDIA.common.dsc.inc b/Platform/NVIDIA/NVIDIA.common.dsc.inc
index 0934c43e..f869f3a7 100644
--- a/Platform/NVIDIA/NVIDIA.common.dsc.inc
+++ b/Platform/NVIDIA/NVIDIA.common.dsc.inc
@@ -543,6 +543,11 @@
 
   gArmTokenSpaceGuid.PcdMmBufferSize|65536
 
+  #
+  # Disable test key detection
+  #
+  gFmpDevicePkgTokenSpaceGuid.PcdFmpDeviceTestKeySha256Digest|{0}
+
 [PcdsFixedAtBuild]
   #
   # Variable Services PCDs.

@KevinFFF - thank you for looking into this!

I am curious about the above patch - I’m certain it works, but it looks like it merely zeros out the hash of the test key.

It is not clear to me that the test key is properly disabled, and that only device keys from the UEFI command line options are being used!

Can you please verify the above?

Thank you!

I’ve also verified that locally.
Configuring PcdFmpDeviceTestKeySha256Digest as zero would disable the compiled in test key.
If keys are in variable store, the test key should not be compiled.

Thank you!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.