Orin AGX no boot, no UART, after flashing secure boot and encrypted internal disk

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
, , ,
1

We are trying to set up Secure boot, Secure UEFI, and encrypted internal disk on our Orin AGX on a custom carrier board.

We are running Jetson Linux r36.3 and flashing from a ubuntu 22.04 host. And we already have been successfully able to flash several modules unencrypted and without secure boot, as well as tested one module with disk encryption (zero filled key).

After creating all the keys and making a new eks image (put into bootloader folder) we wrote the fuses and flashed the board. However, upon restarting the device there is no output on UART and it does not seem to boot, does not come up on ethernet like it does for non-secure builds, power draw is very similar to when in recovery mode, and device shows up as “NVIDIA Corp. APX” in lsusb.

We are wondering if you have any ideas as to what can be wrong when the board ends up in this state.

We write the following fuses: PublicKeyHash, SecureBootKey, OemK1, BootSecurityInfo (0x20b), and SecurityMode (0x1):
sudo ./odmfuse.sh -X fuse_t234.xml -i 0x23 --auth NS our_board_name
And after successfully fusing, just to check that board is alive:
sudo ./odmfuse.sh --test -X fuse_t234.xml -k “pkc.pem” -S “sbk.key” -i 0x23 --auth SBKPKC our_board_name

We have tried several different variants of build and flashing procedures usually within the same Linux_for_Tegra and bootloader(!) folder:

  • Using flash.sh directly (both with internal and mmcblk0p1 and with and without -r after system image has been created by l4t_initrd_flash.sh or l4t_generate_ota_package.sh):
    sudo ROOTFS_ENC=1 ROOTFS_AB=1 ROOTFS_RETRY_COUNT_MAX=3 ROOTFSSIZE=30380392448 ./flash.sh -S 30380392448 -u “pkc.pem” -v “sbk.key” -i “sym2_t234.key” --uefi-keys “uefi_keys/uefi_keys.conf” --uefi-enc “sym_t234.key” our_board_name internal

  • Creating a massflash (using ./tools/kernel_flash/l4t_initrd_flash.sh) and flashing from the ‘mfi’ folder (booted into initrd, but failed to continue as we did not have xusb/xudc set up in the device tree):
    sudo BOARDID=3701 BOARDSKU=0004 FAB=500 BOARDREV=“R.0” CHIPREV=0 CHIP_SKU=“00:00:00:D2” ROOTFS_ENC=1 ROOTFS_AB=1 ROOTFS_RETRY_COUNT_MAX=3 ROOTFSSIZE=30380392448 ./tools/kernel_flash/l4t_initrd_flash.sh -S 30380392448 -u “pkc.pem” -v “sbk.key” -i “sym2_t234.key” --uefi-keys “uefi_keys/uefi_keys.conf” --uefi-enc “sym_t234.key” --no-flash -p “-r --generic-passphrase -c ./bootloader/generic/cfg/flash_t234_qspi_sdmmc_enc_rootfs_ab.xml” --massflash 1 our_board_name internal
    sudo echo “-1” > /sys/module/usbcore/parameters/autosuspend
    sudo systemctl stop udisks2.service
    sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 1

  • Creating a OTA package using ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh and then flashing using the above ./flash.sh with “-r”:
    sudo BOARDID=3701 BOARDSKU=0004 FAB=500 BOARDREV=“R.0” CHIPREV=0 CHIP_SKU=“00:00:00:D2” ROOTFS_ENC=1 ROOTFS_AB=1 ROOTFS_RETRY_COUNT_MAX=3 ROOTFSSIZE=30380392448 ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -s -S 30380392448 -u “pkc.pem” -v “sbk.key” -i “sym2_t234.key” -p “-r -c ./bootloader/generic/cfg/flash_t234_qspi_sdmmc_enc_rootfs_ab.xml --uefi-keys "uefi_keys/uefi_keys.conf" --uefi-enc "sym_t234.key"” our_board_name R36-3

  • Flashing with flashcmd.txt/tegraflash.py directly:
    bootloader$ sudo bash ./flashcmd.txt
    ./tegraflash.py --bl uefi_jetson_with_dtb.bin --bct br_bct_BR.bct --bldtb tegra234-p3701-0000-our_board_name-0000-nv.dtb --applet rcm_2_signed.rcm --applet_softfuse rcm_1_signed.rcm --cmd “secureflash;reboot” --cfg secureflash.xml --chip 0x23 --mb1_bct mb1_bct_MB1_sigheader_encrypt.bct.signed --mem_bct mem_rcm_sigheader_encrypt.bct.signed --mb1_cold_boot_bct mb1_cold_boot_bct_MB1_sigheader_encrypt.bct.signed --mb1_bin mb1_t234_prod_aligned_sigheader_encrypt.bin.signed --psc_bl1_bin psc_bl1_t234_prod_aligned_sigheader_encrypt.bin.signed --mem_bct_cold_boot mem_coldboot_aligned_sigheader_encrypt.bct.signed --bins “psc_fw pscfw_t234_prod_sigheader_encrypt.bin.signed; mts_mce mce_flash_o10_cr_prod_sigheader_encrypt.bin.signed; tsec_fw tsec_t234_sigheader_encrypt.bin.signed; mb2_applet applet_t234_sigheader_encrypt.bin.signed; mb2_bootloader mb2_t234_with_mb2_bct_MB2_sigheader_encrypt.bin.signed; xusb_fw xusb_t234_prod_sigheader_encrypt.bin.signed; pva_fw nvpva_020_sigheader_encrypt.fw.signed; dce_fw display-t234-dce_sigheader_encrypt.bin.signed; nvdec nvdec_t234_prod_sigheader_encrypt.fw.signed; bpmp_fw bpmp_t234-TE990M-A1_prod_sigheader_encrypt.bin.signed; bpmp_fw_dtb tegra234-bpmp-3701-0004-3737-0000_with_odm_sigheader_encrypt.dtb.signed; rce_fw camera-rtcpu-t234-rce_sigheader_encrypt.img.signed; ape_fw adsp-fw_sigheader_encrypt.bin.signed; spe_fw spe_t234_sigheader_encrypt.bin.signed; tos tos-optee_t234_sigheader_encrypt.img.signed; eks eks_t234_sigheader_encrypt.img.signed”–bct_backup

We are able to reflash the board afterwards, at least run the flash procedure and get the same result and also boot into UEFI and initrd using scripts like tegraflash.py. During trying to make this work we have seen several different errors that might be important:

While building massflash:

  • “End sector for APP_ENC, expected at: 119537630, actual: 0
    Error: Return value 4
    Command tegraparser_v2 --generategpt --pt flash.xml.bin
    cp: cannot stat ‘enc_signed/*’: No such file or directory
    cp: cannot stat ‘enc_signed/flash.xml.tmp’: No such file or directory”
    …
    Stat for blob_uefi_jetson_with_dtb.bin failed
    Error: Return value 19
    Command tegrahost_v2 --chip 0x23 0 --generateblob blob.xml blob.bin"
    Tried a work around by symlinking or copying to the file found in bootloader folder which let :
    sudo ln -sf uefi_jetson_with_dtb_aligned_blob_w_bin_sigheader_encrypt.bin.signed uefi_jetson_with_dtb.bin
    And building OTA or using flash.sh created the enc_signed folder with content.

  • “[0047.110] C> LOADER: Could not read binary 21.
    [0047.114] E> Failed to load MB2
    [0047.117] C> Task 0x46 failed (err: 0x27228f37)
    [0047.121] E> Top caller module: MB2_PARAMS, error module: LOADER, reason: 0x37, aux_info: 0x8f
    [0047.130] C> Boot Info Table status dump :
    0111100000111000110111111111000”

  • “FATAL ERROR [FILE=platform/drivers/uphy/uphy-tegra234.c, ERR_UID=2251]: start PLL 8 calibration failed
    …
    ERROR: camera-ip/isp5/isp5.c:2031 [isp5_pm_init] “ERROR: Failed to turn isp1 power on”
    BUG: core/init/init.c:86 [init_all] “*** FIRMWARE INIT FAILED AT LEVEL 95 ***””
    Fixed by removing attempt of initialization of 10gbps PHY from p3701.conf.common (did not get this error on unsecure modules with same p3701 config):
    #ODMDATA=“gbe-uphy-config-22,hsstp-lane-map-3,nvhs-uphy-config-0,hsio-uphy-config-0,gbe0-enable-10g”;
    ODMDATA=“gbe-uphy-config-0,hsstp-lane-map-3,nvhs-uphy-config-0,hsio-uphy-config-0”;

  • “I> dump bct
    I> strt_pg_num(0) num_of_pgs(16) read_buf(0x400719f8)
    E> LOADER: Invalid value Header magic.
    E> Validation failed for 1 copy of BRBCT @ 0
    I> strt_pg_num(512) num_of_pgs(16) read_buf(0x400719f8)
    E> LOADER: Invalid value Header magic.
    E> Validation failed for 2 copy of BRBCT @ 262144
    I> strt_pg_num(1024) num_of_pgs(16) read_buf(0x400719f8)
    E> LOADER: Invalid value Header magic.
    E> Validation failed for 3 copy of BRBCT @ 524288
    I> strt_pg_num(1536) num_of_pgs(16) read_buf(0x400719f8)
    E> LOADER: Invalid value Header magic.
    E> Validation failed for 4 copy of BRBCT @ 786432
    E> NV3P_SERVER: Failed to get address for br bct from nv3p helper.”
    Messed about a lot with this, seemed sometimes to work with using a different br_bct_BR.bct file (e.g. from OTA or massflash build or vice versa)

  • “[0121.337] I> Task: Load MB2/Applet/FSKP
    [0121.341] I> Loading MB2 Applet
    [0121.344] I> Slot: 1
    [0121.347] E> BLOCK_DEV: Failed to open blockdev.
    [0121.352] E> LOADER: Failed to open blockdev 0(0).
    [0121.357] E> LOADER: Failed to get storage info for binary 21 from loader.
    [0121.364] C> LOADER: Could not read binary 21.”

Other things that might be important:

  • We have tried fusing and flashing this secure boot setup on two different Orin modules and they have different revision number: L.0 and R.0
    Also they seem to report (when queried by flash.sh) different values for BOARDREV, CHIPREV, and CHIP_SKU:
    ./chkbdinfo -i cvm.bin
    3701
    ./chkbdinfo -k cvm.bin
    0004
    ./chkbdinfo -f cvm.bin
    500
    ./chkbdinfo -r cvm.bin
    L.0
    ./chkbdinfo -C cvm.bin
    00:FE:00:02
    From massflash flash-only log:
    “[ 17.3102 ] tegrarcm_v2 --chip 0x23 0 --reboot recovery
    [ 17.3111 ] MB2 Applet version 01.00.0000
    Board ID(3701) version(500) sku(0004) revision(L.0)
    Chip SKU(00:00:00:D2) ramcode(00:00:00:00) fuselevel(fuselevel_production) board_FAB(500)”
    We have tried with BOARDID=3701 BOARDSKU=0004 FAB=500 and BOARDREV=L.0 or BOARDREV=R.0 CHIPREV=0 or CHIPREV=1 CHIP_SKU=00:FE:00:02 or CHIP_SKU=00:00:00:D2 when building/flashing.

We have as you can see created a custom partition flash.xml:
flash_t234_qspi_sdmmc_enc_rootfs_ab.xml based on the encrypted A/B xml, with two additional userdata partitions (encrypt, not reencrypt) just before the UDA partition. And decreased the rootfs partition size accordingly (both A and B)
The our_board_name.conf file has the following flags added from working non-secure setup:
rootfs_ab=1;
#EMMC_CFG=“flash_t234_qspi_sdmmc_rootfs_ab.xml”;
disk_enc_enable=1;
EMMC_CFG=“flash_t234_qspi_sdmmc_enc_rootfs_ab.xml”;

We will try to create a new minimal setup (new Linux_for_Tegra folder ;) ) using only flash.sh directly (not no-flash) to try to make a reproducible log, but appreciate if you have any ideas while we do. :)

2

Hi,
If the device cannot be flashed/booted, please refer to the page to get uart log from the device:
Jetson/General debug - eLinux.org
And get logs of host PC and Jetson device for reference. If you are using custom board, you can compare uart log of developer kit and custom board to get more information.
Also please check FAQs:
Jetson AGX Orin FAQ
If possible, we would suggest follow quick start in developer guide to re-flash the system:
Quick Start — NVIDIA Jetson Linux Developer Guide 1 documentation
And see if the issue still persists on a clean-flashed system.
Thanks!

4

hello asmund,

you’ve enable Jetson security with… bootloader SecureBoot + UEFI SecureBoot + Disk Encryption.
since it’s AGX Orin, let’s test re-flashing with flash.sh directly. please setup a serial console, and gather the complete logs for reference.

BTW, please see-also Topic 300161 for checking bit-9 of FUSE_BOOT_SECURITY_INFO_0 .

5

Created clean Linux_for_Tegra folder and then ran the following flash.sh command:
sudo ROOTFS_ENC=1 ROOTFS_AB=1 ROOTFS_RETRY_COUNT_MAX=3 ROOTFSSIZE=30380392448 ./flash.sh -S 30380392448 -u “ecp521.pem” -v “sbk.key” -i “sym2_t234.key” --uefi-keys “uefi_keys/uefi_keys.conf” --uefi-enc “sym_t234.key” my_board_name internal

Realized while it was generating system.img that udisks2 service was activated again, so deactivated that (systemctl stop udisks2.service). Reran the flash.sh command a second time just in case this had caused any problems. Same error:
“[ 7.2039 ] tegraparser_v2 --generategpt --pt flash.xml.bin
[ 7.2042 ] gpt_secondary_3_0.bin:
…
[ 7.2058 ] End sector for APP_ENC_b, expected at: 119537630, actual: 0
[ 7.2058 ]
Error: Return value 4
Command tegraparser_v2 --generategpt --pt flash.xml.bin
cp: cannot stat ‘enc_signed/*’: No such file or directory
cp: cannot stat ‘enc_signed/flash.xml.tmp’: No such file or directory
sed: can’t read flash.xml.tmp: No such file or directory
Using bpmp-dtb concatenated with odmdata in blob for t23x
secureflash.xml:1: parser error : Document is empty”

Also ran odmreadfuse.sh afterwards:
sudo ./odmfuseread.sh -i 0x23 -k ecp521.pem -S sbk.key my_board_name

but it fails while trying to load signed MB1 (mb1_t234_prod_aligned_sigheader_encrypt.bin.signed does not exist in any subdirectory of Linux_for_Tegra, but I do have it in my jumbled together Linux_for_Tegra folder, I believe building OTA package might have created those files):
“[ 0.4117 ] File mb1_t234_prod_aligned_sigheader_encrypt.bin.signed open failed
Error: Return value 19”

Logs include output for all three of these commands (note nothing out on UART for odmreadfuse.sh)

Since the flash.sh command fails before attempting to actually write to the card I suspect something I did to work around this error and successfully flash (something) must be the fault.
2024-11-15_flash_uvi_uart_redacted.txt (10.5 KB)
2024-11-15_flash_uvi_host_redacted.txt (326.5 KB)

6

hello asmund,

is this AGX Orin developer kit?

let’s narrow down the issue, please try following to add only PKC/SBK keys for checking flash progress.
it shall booting up till bootloader stage, and reporting invalidate keys failure.
for instance,
$ sudo ./flash.sh -u “ecp521.pem” -v “sbk.key” jetson-agx-orin-devkit mmcblk0p1

7

Tried, as suggested, to flash without disk encryption
- Edited myboardname.conf to remove enc_disk=1 and use flash_sdmmc… file without disk encryption
- Running command

sudo ROOTFS_AB=1 ROOTFS_RETRY_COUNT_MAX=3 ROOTFSSIZE=30380392448 ./flash.sh -S 30380392448 -u ecp521.pem -v sbk.key --uefi-keys uefi_keys/uefi_keys.conf --uefi-enc sym_t234.key my_board_name mmcblk0p1
  • Same error about size of partition and missing enc_signed(!) folder and files
  • Still error on missing MB2 with cold boot bct failed
    “[ 201.4960 ] File to be written cannot be of zero size, pFileName=mb2_t234_with_mb2_cold_boot_bct_MB2.bin
    Error: Return value 10
    Command tegradevflash_v2 --pt secureflash.xml.bin --create”

Tried on a whim, or inspired by your suggestion of a simpler flash.sh command, to use smaller (halved size) rootfs size:

sudo ROOTFS_ENC=1 ROOTFS_AB=1 ROOTFS_RETRY_COUNT_MAX=3 ROOTFSSIZE=15190196224 ./flash.sh -S 15190196224 -u ecp521.pem -v sbk.key --uefi-keys uefi_keys/uefi_keys.conf --uefi-enc sym_t234.key my_board_name internal
  • Flash successful!
  • No complaints about missing MB2
  • No complaints about missing enc_signed folder
  • No complaints about expected end of sector for APP_b

So “expected large size, got 0” error looks to be because of wrong ROOTFSSIZE variable, decreasing it worked (but I had to decrease it by 1100Mb from 30380392448 to 29226958848! This also seems to leave some unused space on the device).

$ parted /dev/mmcblk0
GNU Parted 3.4
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: MMC G1M15M (sd/mmc)
Disk /dev/mmcblk0: 63.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name                Flags
 5      20.5kB  134MB   134MB                A_kernel            msftdata
 6      134MB   135MB   786kB                A_kernel-dtb        msftdata
 7      135MB   168MB   33.2MB               A_reserved_on_user  msftdata
 8      168MB   302MB   134MB                B_kernel            msftdata
 9      302MB   303MB   786kB                B_kernel-dtb        msftdata
10      303MB   336MB   33.2MB               B_reserved_on_user  msftdata
11      336MB   420MB   83.9MB               recovery            msftdata
12      420MB   421MB   524kB                recovery-dtb        msftdata
13      421MB   488MB   67.1MB  fat32        esp                 boot, esp
14      488MB   572MB   83.9MB               recovery_alt        msftdata
15      572MB   572MB   524kB                recovery-dtb_alt    msftdata
16      572MB   639MB   67.1MB               esp_alt             msftdata
17      639MB   849MB   210MB   ext4         log                 msftdata
18      849MB   1373MB  524MB   ext4         data                msftdata
19      1373MB  1793MB  419MB                UDA                 msftdata
20      1793MB  2296MB  503MB                reserved            msftdata
 1      2296MB  2715MB  419MB   ext4         APP                 msftdata
 2      2715MB  3134MB  419MB   ext4         APP_b               msftdata
 3      3134MB  31.9GB  28.8GB               APP_ENC             msftdata
 4      31.9GB  60.7GB  28.8GB               APP_ENC_b           msftdata

Last End entry 60.7GB, while disk should fit 63.7GB

8

Flashing a board from scratch we see the same problem described in Unable to fuse Orin NX where we fail to burn fuses.
Following that post instructions, we tried to flash the board first. As the board does not have any fuses burned (flashing would fail with -u ecp521.pem -v sbk.key flags) we flashed with non-secure bootloaders:

sudo ROOTFS_ENC=1 ROOTFS_AB=1 ROOTFS_RETRY_COUNT_MAX=3 ROOTFSSIZE=29226958848 ./flash.sh -S 29226958848 -i sym2_t234.key --uefi-keys uefi_keys/uefi_keys.conf --uefi-enc sym_t234.key my_board_name internal

After doing so, and putting board in recovery mode, we are able to burn fuses:

sudo ./odmfuse.sh -X fuse_t234.xml -i 0x23 --auth NS my_board_name

However, afterwards the board does not boot and nothing is printed on UART.
This is probably what we saw before (title of thread) and the reason is probably the board trying to check bootloader signatures and rejecting them since we wrote non-secure bootloaders to be able to burn fuses?

Flashing board again with -u ecp521.pem -v sbk.key flags succeeds and board boots. :)

This adds a lot of time to our flashing process and I will make another post on the subject.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.