Orin Nano experiences probabilistic boot failure after enabling secureboot

Hi Nvidia Team:

I am currently working with the R35.3.1 BSP. My device Orin Nano 8G boots up fine with the following fuse configuration:

<genericfuse MagicId="0x45535546" version="1.0.0">
    <fuse name="OdmId" size="8" value="0x1000000000000001"/>
    <fuse name="OdmInfo" size="4" value="0x10000001"/>
    <fuse name="ArmJtagDisable" size="4" value="0x1"/>
    <fuse name="DebugAuthentication" size="4" value="0x1F"/>
    <fuse name="CcplexDfdAccessDisable" size="4" value="0x1"/>
    <fuse name="ZeroizeDis" size="4" value="0x1"/>
    <fuse name="PublicKeyHash" size="64" value="******"/>
    <fuse name="PkcPubkeyHash1" size="64" value="******"/>
    <fuse name="PkcPubkeyHash2" size="64" value="******"/>
    <fuse name="SecureBootKey" size="32" value="******"/>
    <fuse name="OemK1" size="32" value="******"/>
    <fuse name="OemK2" size="32" value="******"/>
    <fuse name="Kdk0" size="32" value="******"/>
    <fuse name="BootSecurityInfo" size="4" value="0x2be9"/>
    <fuse name="SecurityMode" size="4" value="0x1"/>
</genericfuse>

and the flash script:

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

BASE_ARGS=(
    BOARDID=3767
    BOARDSKU=0003
    FAB=300
    CHIP_SKU="00:00:00:D5"
)

# Args shared by all steps (keys, board, rootdev)
COMMON_ARGS=(
    --showlogs
    -u bootloader/security/orin_nano/rsa3k-0.pem
    -v bootloader/security/orin_nano/sbk.key
    --network usb0
    jetson-orin-nano-devkit-nvme
    internal
)

# Step 3/5: external (NVMe) device args
EXTERNAL_ARGS=(
    --external-device nvme0n1p1
    -S 14GiB
    -c ./tools/kernel_flash/flash_l4t_external.xml
)

echo ""
echo "=== Step 3: Generate all images (QSPI + NVMe) ==="
sudo env "${BASE_ARGS[@]}" \
    ./tools/kernel_flash/l4t_initrd_flash.sh \
    --no-flash \
    "${EXTERNAL_ARGS[@]}" \
    "${COMMON_ARGS[@]}"

echo ""
echo "=== Step 6: Flash device (--flash-only) ==="
sudo env "${BASE_ARGS[@]}" \
    ./tools/kernel_flash/l4t_initrd_flash.sh \
    --flash-only \
    "${EXTERNAL_ARGS[@]}" \
    "${COMMON_ARGS[@]}"

Currently, there is a 50% chance of getting stuck at the MB2 stage and encountering the following error:

I> KDF derivation: OK
I> KDF derivation: OK
I> GCM decryption: OK
I> cpubl: Authentication Finalize Done
I> Binary cpubl loaded successfully at 0x268200000
I> RSA PSS signature check: OK
I> KDF derivation: OK
I> KDF derivation: OK
I> GCM decryption: OK
I> tos: Authentication Finalize Done
I> Binary tos loaded successfully at 0x277dfe000
I> KDF derivation: OK
I> ECDSA P256 signing: OK
I> Relocating OP-TEE dtb from: 0x277eff770 to 0x268630000, size: 0x1976
I> [0] START: 0x80000000, SIZE: 0x1e8670000
I> [1] START: 0x26e000000, SIZE: 0x2000000
I> Setting NS memory ranges to OP-TEE dtb finished.
I> Partition name: A_eks
I> Size of partition: 262144
I> Binary@ device:3/0 block-44288 (partition size: 0x40000), name: A_eks
E> Heap free list corrupted !!!
鐳CE: FW Boot Done
錥     2.948348] Camera-FW on t234-rce-safe ready SHA1=97e50cbf (crt 12.945 ms, total boot 233.523 ms)

While the remaining 50% chance startup normally and entry into the system.

No reprogramming was conducted during this period. I have attached the logs of successful and failed startups. Could you please help me identify the reason, thanks.

boot_failed.txt (27.0 KB)

boot_success.txt (93.8 KB)

it looks like a follow-up of Topic 385344.

hello lingsong.zheng,

it looks the failure caused by EKS image loading path.
EKS contains an Encrypted Keyblob (EKB), which carries user-defined secrets such as encryption keys. MB2 loads the blob from A_eks and passes it to OP-TEE. OP-TEE separately authenticates and decrypts its contents using keys derived from the OEM fuse key.

here’re some questions for your examination,
(1) how’s your EKS image from? is it the BSP default, generated for R35.3.1, or copied from R39.2?
(2) which release’s gen_ekb.py did you used?

Hi Jerry:

I tried using both the default eks_t234.img and the eks_t234.img generated based on the script in r35.3.1 public_source, but both exhibited the same phenomenon

hello lingsong.zheng,

just curious.. is it the same device but having 50% chance of boot failure?

Hi Jerry:

Yes, I have tested two modules, and both exhibit the same phenomenon. It’s not strictly a 50% failure to boot. Furthermore, both boards boot successfully 100% of the time using the r39.2 BSP. I tested using the r35.6.5 BSP and have not found any similar issues so far. I want to know if it’s possible to resolve on r35.3.1.

hello lingsong.zheng,

all right, thanks for sharing the test results with JP-7.2/r39.2

assume this is known issue in the early JP-5.1.1
as you can see from NVIDIA Jetson Linux 35.3.1, you may give it a try to apply [overlay_mb1bct_35.x.tbz2] for confirmation.
this overlay addresses a QSPI read-timing margin issue.

anyways, may I know what’s the strong request to stay-on JP-5.1.1/r35.3.1 instead of moving forward?

Hi Jerry:

This patch has already been applied, but the issue still persists.

Currently, r35.3.1 is our stable shipping version, while r39.2 is still under development. We need to add the secureboot feature to the version we are currently shipping. Once r39.2 is developed, we will uniformly switch to the r39.2 version

hello lingsong.zheng,

BTW,
here’s one thing you should aware, those no online approach to enable bootloader secureboot for those shipped targets.
please see-also Topic 365443, it’s a must to put device enter forced-recovery mode, and having wired flashing process (a full-flash) to enable bootloader secure boot.

Hi Jerry:

I am aware of this situation, which is why we intend to program the fuse on r35.3 before shipping

hello lingsong.zheng,

it’s not suggest to stay-on r35.3.1
please refer to developer guide, Image-Based Over-the-Air Update.
you should at least based-on r35.5.0 for updating to r39.2.1 for OTA scenario.

Hi Jerry:

OK, we’ll do it on r35.6.5.