MB1 BL crash when rebooting to RCM from B slot

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
, ,
1

On an A/B-Slot image with B slot booted, when rebooting to RCM recovery (sudo reboot forced-recovery) and then attempting to flash, the MB1 crashes while trying to load the MB2 Applet. While I am currently using Jetpack 5 via yocto/meta-tegra, I was also able to reproduce the issue with Jetpack 6/Jetson Linux R36.4 and the sample rootfs all as directly provided by nvidia.

Steps with which I can reproduce the issue:

  1. Flash Jetson Orin Nano Devkit with A/B slots enabled with the sample rootfs of Jetson Linux R36.4
  2. Boot device and do initial configuration
  3. sudo nvbootctrl set-active-boot-slot 1, reboot
  4. Do initial configuration again
  5. sudo nvbootctrl dump-slots-info:
Current version: 36.4.0
Capsule update status: 0
Current bootloader slot: B
Active bootloader slot: B
num_slots: 2
slot: 0,             status: normal
slot: 1,             status: normal
  1. reboot forced-recovery
  2. Try to flash from PC: sudo ./flash.sh jetson-orin-nano-devkit mmcblk0p1
Flash Log 
###############################################################################
# L4T BSP Information:
# R36 , REVISION: 4.0
# User release: 0.0
./flash.sh: line 1989: /home/pkauls/tegra/Linux_for_Tegra/rootfs/etc/user_release_version: No such file or directory
###############################################################################
ECID is 0x80012344705DD3570800000001028140
# Target Board Information:
# Name: jetson-orin-nano-devkit, Board Family: generic, SoC: Tegra 234,
# OpMode: production, Boot Authentication: NS,
# Disk encryption: disabled ,
###############################################################################
copying device_config(/home/pkauls/tegra/Linux_for_Tegra/bootloader/generic/BCT/tegra234-mb1-bct-device-p3767-0000.dts)... done.
copying misc_config(/home/pkauls/tegra/Linux_for_Tegra/bootloader/generic/BCT/tegra234-mb1-bct-misc-p3767-0000.dts)... done.
copying emc_fuse_dev_params(/home/pkauls/tegra/Linux_for_Tegra/bootloader/generic/BCT/tegra234-br-bct-diag-boot.dts)... done.
Existing emcfuse(/home/pkauls/tegra/Linux_for_Tegra/bootloader/fuse_t234.xml) reused.
./tegraflash.py --chip "0x23" --applet "/home/pkauls/tegra/Linux_for_Tegra/bootloader/mb1_t234_prod.bin" --skipuid --cfg readinfo_t234_min_prod.xml --dev_params tegra234-br-bct-diag-boot.dts --device_config tegra234-mb1-bct-device-p3767-0000.dts --misc_config tegra234-mb1-bct-misc-p3767-0000.dts --bins "mb2_applet applet_t234.bin" --cmd "readfuses fuse_t234.bin fuse_t234.xml; dump eeprom cvm cvm.bin; dump try_custinfo custinfo_out.bin; reboot recovery"
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands

[   0.1803 ] Reading fuses
[   0.1825 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.1843 ] File rcm_state open failed
[   0.1843 ] ERROR: failed to read rcm_state
[   0.1843 ]
[   0.1856 ] tegrasign_v3.py --key None --getmode mode.txt
[   0.1857 ] Assuming zero filled SBK key
[   0.1846 ] Pre-processing config: tegra234-mb1-bct-device-p3767-0000.dts
[   0.2448 ] Pre-processing config: tegra234-mb1-bct-misc-p3767-0000.dts
[   0.2550 ] Parsing partition layout
[   0.2563 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.2591 ] Kernel DTB used: None
[   0.2591 ] WARNING: dce base dtb is not provided

[   0.2591 ] Parsing partition layout
[   0.2596 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.2609 ] Creating list of images to be signed
[   0.2620 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --list images_list.xml zerosbk
[   0.2632 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.2654 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.2659 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.2701 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.2708 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.2745 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.2747 ] adding BCH for mb2_t234_aligned.bin
[   0.2768 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.2831 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.2834 ] adding BCH for mb2_t234_aligned.bin
[   0.2918 ] Filling MB1 storage info
[   0.2919 ] Parsing dev params for multi chains
[   0.2972 ] Generating br-bct
[   0.2992 ] Updating dev and MSS params in BR BCT
[   0.2993 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.3044 ] Updating bl info
[   0.3055 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin
[   0.3062 ] WARNING: boot chain is not completed. set to 0
[   0.3070 ] Generating signatures
[   0.3083 ] tegrasign_v3.py --key None --list images_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.3084 ] Assuming zero filled SBK key
[   0.3288 ] Warning: pub_key.key is not found
[   0.3276 ] Parsing dev params for multi chains
[   0.3276 ] Generating br-bct
[   0.3287 ] Updating dev and MSS params in BR BCT
[   0.3287 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.3295 ] Updating bl info
[   0.3309 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.3317 ] WARNING: boot chain is not completed. set to 0
[   0.3334 ] Generating SHA2 Hash
[   0.3364 ] Sha saved in br_bct_BR.sha
[   0.3352 ] Get Signed section of bct
[   0.3357 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --listbct bct_list.xml
[   0.3384 ] Signing BCT
[   0.3397 ] tegrasign_v3.py --key None --list bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.3398 ] Assuming zero filled SBK key
[   0.3435 ] Sha saved in br_bct_BR.sha
[   0.3436 ] Warning: pub_key.key is not found
[   0.3424 ] Updating BCT with signature
[   0.3433 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesig bct_list_signed.xml
[   0.3456 ] Offset :4608 Len :3584
[   0.3463 ] Generating SHA2 Hash
[   0.3477 ] tegrasign_v3.py --key None --list bct_list.xml --sha sha512
[   0.3478 ] Assuming zero filled SBK key
[   0.3478 ] Assuming zero filled SBK key
[   0.3508 ] Sha saved in br_bct_BR.sha
[   0.3497 ] Updating BCT with SHA2 Hash
[   0.3507 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesha bct_list_signed.xml
[   0.3514 ] Offset :4608 Len :3584
[   0.3517 ] Offset :68 Len :8124
[   0.3524 ] Generating coldboot mb1-bct
[   0.3533 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct.cfg --misc tegra234-mb1-bct-misc-p3767-0000_cpp.dtb --device tegra234-mb1-bct-device-p3767-0000_cpp.dtb
[   0.3540 ] MB1-BCT version: 0.13

[   0.3560 ] Parsing config file :tegra234-mb1-bct-device-p3767-0000_cpp.dtb
[   0.3560 ] Added Platform Config 9 data with size :- 100
[   0.3560 ]
[   0.3560 ] Updating mb1-bct with firmware information
[   0.3566 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct_MB1.bct --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.3590 ] tegrahost_v2 --chip 0x23 0 --align mb1_cold_boot_bct_MB1_aligned.bct
[   0.3596 ] Generating SHA2 Hash for mb1bct
[   0.3620 ] Sha saved in mb1_cold_boot_bct_MB1_aligned.sha
[   0.3632 ] Sha saved in mb1_cold_boot_bct_MB1.sha
[   0.3629 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_cold_boot_bct_MB1_aligned.bct zerosbk
[   0.3640 ] adding BCH for mb1_cold_boot_bct_MB1_aligned.bct
[   0.3657 ] tegrasign_v3.py --key None --list mb1_cold_boot_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.3658 ] Assuming zero filled SBK key
[   0.3679 ] Warning: pub_key.key is not found
[   0.3673 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_cold_boot_bct_MB1_aligned_sigheader.bct.encrypt mb1_cold_boot_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.3683 ] Generating recovery mb1-bct
[   0.3688 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct.cfg --misc tegra234-mb1-bct-misc-p3767-0000_cpp.dtb --device tegra234-mb1-bct-device-p3767-0000_cpp.dtb
[   0.3695 ] MB1-BCT version: 0.13

[   0.3713 ] Parsing config file :tegra234-mb1-bct-device-p3767-0000_cpp.dtb
[   0.3713 ] Added Platform Config 9 data with size :- 100
[   0.3713 ]
[   0.3713 ] Updating mb1-bct with firmware information
[   0.3717 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct_MB1.bct --recov --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.3738 ] tegrahost_v2 --chip 0x23 0 --align mb1_bct_MB1_aligned.bct
[   0.3750 ] Generating SHA2 Hash for mb1bct
[   0.3773 ] Sha saved in mb1_bct_MB1_aligned.sha
[   0.3793 ] Sha saved in mb1_bct_MB1.sha
[   0.3803 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_bct_MB1_aligned.bct zerosbk
[   0.3820 ] adding BCH for mb1_bct_MB1_aligned.bct
[   0.3845 ] tegrasign_v3.py --key None --list mb1_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.3846 ] Assuming zero filled SBK key
[   0.3860 ] Warning: pub_key.key is not found
[   0.3853 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_bct_MB1_aligned_sigheader.bct.encrypt mb1_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.3864 ] Info: Skip generating mem_bct because sdram_config is not defined
[   0.3864 ] Info: Skip generating mem_bct because sdram_config is not defined
[   0.3864 ] Copying signatures
[   0.3873 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.3953 ] mb1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.3954 ] psc_bl1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.3954 ] Boot Rom communication
[   0.3960 ] tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[   0.3968 ] BR_CID: 0x80012344705DD3570800000001028140
[   0.3990 ] Sending bct_br
[   0.3991 ] Sending mb1
[   0.4003 ] Sending psc_bl1
[   0.4282 ] Sending bct_mb1
[   0.4424 ] Boot Rom communication completed
[   0.4457 ] tegrahost_v2 --chip 0x23 0 --align applet_t234_aligned.bin
[   0.4472 ] tegrahost_v2 --chip 0x23 0 --magicid MB2A --appendsigheader applet_t234_aligned.bin zerosbk
[   0.4475 ] adding BCH for applet_t234_aligned.bin
[   0.4525 ] tegrasign_v3.py --key None --list applet_t234_aligned_sigheader.bin_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.4526 ] Assuming zero filled SBK key
[   0.4548 ] Warning: pub_key.key is not found
[   0.4542 ] tegrahost_v2 --chip 0x23 0 --updatesigheader applet_t234_aligned_sigheader.bin.encrypt applet_t234_aligned_sigheader.bin.hash zerosbk
[   0.4556 ] Sending mb2_applet...

[   0.4561 ] tegrarcm_v2 --chip 0x23 0 --pollbl --download applet applet_t234_sigheader.bin.encrypt
[   0.4571 ] BL: version 1.4.0.4-t234-54845784-e89ea9bc last_boot_error: 0
[   0.6667 ] Sending applet
[   0.6668 ] ERROR: might be timeout in USB write.
Error: Return value 3
Command tegrarcm_v2 --chip 0x23 0 --pollbl --download applet applet_t234_sigheader.bin.encrypt
[   0.7648 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.7657 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.7664 ] Retrieving board information
[   0.7670 ] tegrarcm_v2 --chip 0x23 0 --oem platformdetails chip chip_info.bin
[   0.7686 ] Retrieving EEPROM data
[   0.7687 ] tegrarcm_v2 --oem platformdetails eeprom cvm /home/pkauls/tegra/Linux_for_Tegra/bootloader/cvm.bin --chip 0x23 0
[   0.7727 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.7750 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.7763 ] Dumping customer Info
[   0.7774 ] tegrarcm_v2 --chip 0x23 0 --oem dump bct tmp.bct
[   0.7790 ] tegrabct_v2 --brbct tmp.bct --chip 0x23 0 --custinfo /home/pkauls/tegra/Linux_for_Tegra/bootloader/custinfo_out.bin
[   0.7798 ] File tmp.bct open failed
[   0.7798 ]
[   0.7798 ] Error: try getting custinfo fail. Moving on
[   0.7798 ] Rebooting to recovery mode
[   0.7817 ] tegrarcm_v2 --chip 0x23 0 --ismb2
[   0.7826 ] Rebooting to recovery mode
[   0.7831 ] tegrarcm_v2 --chip 0x23 0 --reboot recovery
Parsing board information failed.
UART Log 
[0049.473] I> MB1 (version: 1.4.0.1-t234-54845784-08e631ca)
[0049.478] I> t234-A01-0-Silicon (0x12347) Prod
[0049.482] I> Boot-mode : BPMP Diagnostic
[0049.486] I> Entry timestamp: 0x00000000
[0049.490] I> last_boot_error: 0x0
[0049.493] I> BR-BCT: preprod_dev_sign: 0
[0049.497] I> rst_source: 0xb, rst_level: 0x1
[0049.501] I> Task: Initialize MB2 params
[0049.505] I> MB2-params @ 0x40060000
[0049.509] I> Task: Crypto init
[0049.512] I> Task: Perform MB1 KAT tests
[0049.516] I> Task: NVRNG health check
[0049.519] I> NVRNG: Health check success
[0049.523] I> Task: Secure debug controls
[0049.527] I> Task: strap war set
[0049.530] I> Task: Program NV master stream id
[0049.534] I> Task: Verify boot mode
[0049.540] I> Task: Alias fuses
[0049.543] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0049.550] I> Task: Print SKU type
[0049.553] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x000001c8
[0049.558] I> FUSE_OPT_GPC_DISABLE = 0x00000002
[0049.563] I> FUSE_OPT_TPC_DISABLE = 0x000000f0
[0049.567] I> FUSE_OPT_DLA_DISABLE = 0x00000003
[0049.571] I> FUSE_OPT_PVA_DISABLE = 0x00000001
[0049.575] I> FUSE_OPT_NVENC_DISABLE = 0x00000001
[0049.580] I> FUSE_OPT_NVDEC_DISABLE = 0x00000000
[0049.584] I> FUSE_OPT_FSI_DISABLE = 0x00000001
[0049.589] I> FUSE_OPT_EMC_DISABLE = 0x0000000c
[0049.593] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0049.597] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0049.601] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0049.605] I> FUSE_SKU_INFO_0 = 0xd5
[0049.609] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS
[0049.613] I> FUSE_PACKAGE_INFO_0 = 0x2
[0049.616] I> SKU: Prod
[0049.619] I> Task: Boost clocks
[0049.622] I> Initializing PLLC2 for AXI_CBB.
[0049.626] I> AXI_CBB : src = 35, divisor = 0
[0049.630] I> Task: Voltage monitor
[0049.633] I> VMON: Vmon re-calibration and fine tuning done
[0049.639] I> Task: UPHY init
[0049.642] W> UPHY: UPHY lane info table is empty in MB1 BCT.
[0049.648] I> Task: Boot device init
[0049.651] I> Boot_device: RCM
[0049.654] I> USB configuration success
[0049.658] I> Task: TSC init
[0049.661] I> Task: Enable WDT 5th expiry
[0049.665] I> Task: I2C register
[0049.668] I> Task: Set I2C bus freq
[0049.671] I> Task: Reset FSI
[0049.674] I> Task: Enable clock-mon
[0049.678] I> FMON: Fmon re-programming done
[0049.682] I> Task: Mapper init
[0049.685] I> Task: MB1 fixed firewalls
[0049.691] W> Firewall readback mismatch
[0049.696] I> Task: Load MB2/Applet/FSKP
[0049.700] I> Loading MB2 Applet
[0049.703] I> Slot: 1
[0049.705] E> BLOCK_DEV: Failed to open blockdev.
[0049.710] E> LOADER: Failed to open blockdev 0(0).
[0049.715] E> LOADER: Failed to get storage info for binary 21 from loader.
[0049.723] C> LOADER: Could not read binary 21.
[0049.727] E> Failed to load MB2
[0049.730] C> Task 0x46 failed (err: 0x27228311)
[0049.734] E> Top caller module: MB2_PARAMS, error module: LOADER, reason: 0x11, aux_info: 0x83
[0049.743] C> Boot Info Table status dump :
0111100000111000110111111111000000011110000000000000011000001

This only happens when rebooting from a system running on slot B. Running slot A, setting slot B active, rebooting to RCM works.

3

Hi patrick.kauls,

Could you share the result of the following command on your board?

$ cat /etc/nv_boot_control.conf

Do you mean only boot-chain A/B or also Rootfs A/B enabled?

Please share the command you used to flash the devkit in 1st step.

Could you run the following command instead to check?

$ sudo nvbootctrl -t rootfs dump-slots-info
4

It is not an expected command to flash Orin Nano.

Have you connected a SD or NVMe SSD on your board?

5 
TNSPEC 3767-300-0005-L.1-1-1-jetson-orin-nano-devkit-
COMPATIBLE_SPEC 3767--0005--1--jetson-orin-nano-devkit-
TEGRA_BOOT_STORAGE nvme0n1
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

Rootfs A/B enabled. I did not try with just boot-chain A/B though.

Taken from the quick start guide with only edits to enable A/B rootfs:
$ sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_t234_nvme_rootfs_ab.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal

Current rootfs slot: B
Active rootfs slot: B
num_slots: 2
slot: 0,             retry_count: 3,             status: normal
slot: 1,             retry_count: 3,             status: normal

I just retested with the initial flashing command as stated above, with the same result. Since the failure is already during the first load of MB2 applet, I don’t think the specific command matters much though.
The board has an SSD connected, though I was able to reproduce the same behavior on another devkit running an older version of Linux for Tegra on an SD card. (That one was flashed a while ago though, so I don’t have the commands used there.)

6

Boot-chain A/B is enabled by default.

Do you mean the failure happen when it is booting from boot-chain B?
Would it work if you perform the following steps?

1. Flash Jetson Orin Nano Devkit with A/B slots enabled with the sample rootfs of Jetson Linux R36.4
2. Boot device and do initial configuration
3. sudo nvbootctrl set-active-boot-slot 1, reboot
4. Do initial configuration again
5. sudo nvbootctrl dump-slots-info
6. reboot
7. sudo reboot forced-recovery
8. sudo ./tools/kernel_flash/l4t_initrd_flash.sh --initrd jetson-orin-nano-devkit internal
7

Yes. Specifically, the currently booted system must use boot-chain B for it to fail, the active slot in nvbootctrl does not seem to matter even if changed before the reboot to recovery:

  • Booting from slot A, not changing boot slot → works
  • Booting from slot A, set-active-boot-slot 1 → works
  • Booting from slot B, not changing boot slot → fails
  • Booting from slot B, set-active-boot-slot 0 → fails

No, I tested and this also does not work. Intermittent clean reboots do not change anything.

8

Any update on this? Is there maybe a known workaround? Using the hardware RCM pin unfortunately is not an option on our custom board.

Our current workaround seems not very practicable (reboot into slot A, if that does not work firmware update a working firmware into slot A, reboot, then reboot into RCM).

9

I just tried to verify on my Orin Nano devkit.
I can reproduce the above messages in serial console log.
But I still can flash the QSPI successfully with the following command.

$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh bootloader/generic/cfg/flash_t234_qspi.xml jetson-orin-nano-devkit internal

[   0.6668 ] ERROR: might be timeout in USB write.
Error: Return value 3
..
Parsing board information failed.

From your failed flash log, it seems caused from USB connection.
Please try using another USB cable or host to check if it may help.

10

I have tried several USB cables, but it consistently does not work when going into RCM from Slot A and works when going into RCM from slot A. This to me makes USB signal integrity issues unlikely.

I see the same behavior when using UEFI to reboot into RCM (setting the BoardRecoveryBoot UEFI variable to true, which after a reboot or power cycle has UEFI reboot into RCM) - both with warm reboot and cold boot.

My best guess so far is that the scratch register 109 holding the currently active boot chain is initialized to use chain B when warm booting into RCM, but chain A when cold booting (powerup with recovery pin active). Then, even with the BR and MB1 being downloaded via USB, it does not download the MB2 applet via USB, and instead tries to boot from slot 1.

Is there a way to manually set the scratch register 109 from runtime to debug this further? Unfortunately, all I could find was the layout of the register itself, but the scratch registers are not listed in the TRM.

11

Any news on this? Just to make sure, I was able to reproduce this on multiple devkits as well as custom hardware with multiple Jetpack versions, multiple USB cables and multiple PCs. In no instance was I able to flash anything - it always failed when trying to load the MB2 applet and read out the board information.

12

Would you hit the issue if you put the device into RCM mode from B slot manually?
(i.e. short REC/GND and power up the board to enter into force recovery state before flash)

Could you elaborate on this? How do you determine that the issue is caused from “scratch register 109”?

You have tried R36.4.0. Have you also verified with R35.6.0 on the devkit?

13

No. Shorting the GPIO and doing a hard reset/power cycle always works.

A few days ago I was able to narrow down the cause further and find a workaround, which further points to scratch register 109 (SCR109).

SCR109 bits 4:5 represent the boot chain to use. On coldboot BR reads the current boot slot from BR-BCT into SCR109, on warmboot the value in the register is used as-is. The value in the register is then used by MB1 and MB2 to know which boot chain is used (see Jetpack docs).
When accessing SCR109 manually using devmem, reading the contents shows the expected results:

# boot chain A active
busybox devmem 0x0C3903CC 32 # result: 0x00000000

# boot chain B active
busybox devmem 0x0C3903CC 32 # result: 0x00000010

I can reliably reproduce the problem from both active boot slots by simply setting the register before rebooting into RCM:

# always reboots to working RCM, regardless of current boot slot
busybox devmem 0x0C3903CC 32 0x00000000 && reboot forced-recovery

# always reboots to broken RCM, regardless of current boot slot
busybox devmem 0x0C3903CC 32 0x00000010 && reboot forced-recovery

The failure is exactly the same as if only rebooting to RCM from the B slot (i.e., successfully enter RCM, but MB2 applet loading crashes).

My best guess is that when triggering RCM in hardware (REC/GND), BR goes into RCM before setting SCR109 from BR-BCT, so it is always zero’d. On a warmboot to RCM the register retains its old value (non-zero in boot chain B). The MB1 downloaded via USB then reads the register (see “Slot: 1” printed in MB1 log) and tries to load the MB2/MB2 applet from an invalid location (instead of downloading it via USB as intended).

While this provides a workaround, it relies on devmem being available and able to access the memory. Since our custom hardware does not have external access to the recovery GPIO, rebooting to RCM is the only option for RCM-level operations (e.g., fusing).

Not extensively, but the general test I did (as described in the original post) yielded the same results.

14

Okay, so the issue is specific to use command entering into force recovery state.

Do you mean that if you write 0x0 to register 0x0C3903CC before running the command to enter force recovery state and you would not hit the issue?

15

Yes. Writing 0x00 before running the command leads to never hitting the issue, Writing 0x10 leads to always hitting the issue. This is regardless of currently active or selected boot chain. When on boot chain A/0, the initial value is 0x00 as set by BR, on boot chain B/1 it is 0x10 as set by BR.

16 
[0034.924] I> Slot: 1
[0034.926] E> BLOCK_DEV: Failed to open blockdev.
[0034.931] E> LOADER: Failed to open blockdev 0(0).
[0034.936] E> LOADER: Failed to get storage info for binary 21 from loader.
[0034.944] C> LOADER: Could not read binary 21.
[0034.948] E> Failed to load MB2
[0034.951] C> Task 0x46 failed (err: 0x27228311)
[0034.955] E> Top caller module: MB2_PARAMS, error module: LOADER, reason: 0x11, aux_info: 0x83
[0034.964] C> Boot Info Table status dump :
0111100000111000110111111111000000011110000000000000011000001

okay, I would get above errors as yours if I booted from slot B and run sudo reboot forced-recovery.
Writing SCR109 as 0x0 before entering RCM can help for above errors.

But I still can flash the board successfully if I have that error during flash.
Could you help to verify again and share the full flash and serial console log for further check?
They seem the different issue.

17

I ran the full process again, and indeed was able to flash the board. Looking further into it however I also found why, and it unfortunately is not a way to fix the issue.

Short version

MB2 applet does not start, but reading out device data fails silently and simply reuses previously read data. Trying to flash with a completely fresh extracted jetpack fails, and when removing bootloader/cvm.bin (the cvm eeprom data from the previous run) before flashing it also fails.
Booting initrd and everything else that does not rely on the MB2 applet works, which is why flashing can be successful.

Long version

What happens on the flashing side is the following:
l4t_initrd_flash.sh calls a few layers deep until it reaches the following tegraflash.py call (see log):

./tegraflash.py \
    --chip "0x23" \
    --applet "/path/to/Linux_for_Tegra/bootloader/mb1_t234_prod.bin" \
    --skipuid \
    # [...]
    --bins "mb2_applet applet_t234.bin" \
    --cmd "readfuses fuse_t234.bin fuse_t234.xml; dump eeprom cvm cvm.bin; dump try_custinfo custinfo_out.bin; reboot recovery"

tegraflash.py then calls the handlers for all commands one after the other:

  1. do_readfuses calls into the t234 impl which first sets up BR communication and so on, but fails to boot into MB2 applet, then raises an error (“timeout in USB write”). This gets caught and printed, reading fuses is cancelled, but execution continues with the next command. This behavior is exclusive to readfuses - all others fail early and cancel further command execution if an unexpected error is encountered.
    All other commands think that an MB2 applet is running because tegrarcm_v2 --ismb2applet succeeds.
  2. do_dump calls into the t234 impl which does a tegrarcm_v2 call to read the eeprom. While this call returns success, no eeprom contents are read from the device or written to disk.
  3. do_dump calls into the t234 impl and tries to read the custinfo, which fails, is logged, and ignored.
  4. do_reboot also seems to succeed, but I can not validate if rebooting actually happens.

dump cvm cvm.bin is suppsed to dump the CVM data to bootloader/cvm.bin, but when the device is in the state as described earlier, MB2 applet load fails and no CVM data is read, leaving data from a previous run (if available) intact. This is then used to flash the device.

For obvious reasons, this is not viable to use in production, especially when flashing multiple devices with potentially different SoM variants.

Could you try to reproduce the issue with either removing bootloader/cvm.bin before flashing or when using a fully fresh jetpack release which was not previously used for flashing?

Logs with bootloader/cvm.bin removed before flashing:

Flash Host Log 
Please install the Secureboot package to use initrd flash for fused board
# Entry added by NVIDIA initrd flash tool
/home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/tools/kernel_flash/tmp 127.0.0.1(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
rpcbind: another rpcbind is already running. Aborting
Export list for localhost:
/home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/tools/kernel_flash/tmp 127.0.0.1
/home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/tools/kernel_flash/l4t_initrd_flash_internal.sh --no-flash bootloader/generic/cfg/flash_t234_qspi.xml jetson-orin-nano-devkit internal
************************************
*                                  *
*  Step 1: Generate flash packages *
*                                  *
************************************
Create folder to store images to flash
Generate image for internal storage devices
Generate images to be flashed
ADDITIONAL_DTB_OVERLAY=""  /home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/flash.sh --no-flash --sign  jetson-orin-nano-devkit internal

###############################################################################
# L4T BSP Information:
# R36 , REVISION: 4.0
# User release: 0.0
###############################################################################
ECID is 0x80012344705DD3570800000001028140
copying device_config(/home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/bootloader/generic/BCT/tegra234-mb1-bct-device-p3767-0000.dts)... done.
copying misc_config(/home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/bootloader/generic/BCT/tegra234-mb1-bct-misc-p3767-0000.dts)... done.
copying emc_fuse_dev_params(/home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/bootloader/generic/BCT/tegra234-br-bct-diag-boot.dts)... done.
Existing emcfuse(/home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/bootloader/fuse_t234.xml) reused.
./tegraflash.py --chip "0x23" --applet "/home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/bootloader/mb1_t234_prod.bin" --skipuid --cfg readinfo_t234_min_prod.xml --dev_params tegra234-br-bct-diag-boot.dts --device_config tegra234-mb1-bct-device-p3767-0000.dts --misc_config tegra234-mb1-bct-misc-p3767-0000.dts --bins "mb2_applet applet_t234.bin" --cmd "readfuses fuse_t234.bin fuse_t234.xml; dump eeprom cvm cvm.bin; dump try_custinfo custinfo_out.bin; reboot recovery" 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0195 ] Reading fuses
[   0.0199 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.0201 ] File rcm_state open failed
[   0.0203 ] ERROR: failed to read rcm_state
[   0.0203 ] 
[   0.0210 ] tegrasign_v3.py --key None --getmode mode.txt
[   0.0211 ] Assuming zero filled SBK key
[   0.0206 ] Pre-processing config: tegra234-mb1-bct-device-p3767-0000.dts
[   0.0264 ] Pre-processing config: tegra234-mb1-bct-misc-p3767-0000.dts
[   0.0346 ] Parsing partition layout
[   0.0349 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.0357 ] Kernel DTB used: None
[   0.0358 ] WARNING: dce base dtb is not provided

[   0.0358 ] Parsing partition layout
[   0.0361 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.0366 ] Creating list of images to be signed
[   0.0369 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --list images_list.xml zerosbk
[   0.0372 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.0382 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.0385 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.0430 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.0436 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.0481 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.0482 ] adding BCH for mb2_t234_aligned.bin
[   0.0508 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.0624 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.0625 ] adding BCH for mb2_t234_aligned.bin
[   0.0766 ] Filling MB1 storage info
[   0.0766 ] Parsing dev params for multi chains
[   0.0826 ] Generating br-bct
[   0.0829 ] Updating dev and MSS params in BR BCT
[   0.0829 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.0834 ] Updating bl info
[   0.0838 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin
[   0.0840 ] WARNING: boot chain is not completed. set to 0
[   0.0847 ] Generating signatures
[   0.0853 ] tegrasign_v3.py --key None --list images_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.0854 ] Assuming zero filled SBK key
[   0.0940 ] Warning: pub_key.key is not found
[   0.0935 ] Parsing dev params for multi chains
[   0.0935 ] Generating br-bct
[   0.0938 ] Updating dev and MSS params in BR BCT
[   0.0939 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.0944 ] Updating bl info
[   0.0947 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.0950 ] WARNING: boot chain is not completed. set to 0
[   0.0960 ] Generating SHA2 Hash
[   0.0973 ] Sha saved in br_bct_BR.sha
[   0.0968 ] Get Signed section of bct
[   0.0970 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --listbct bct_list.xml
[   0.0975 ] Signing BCT
[   0.0982 ] tegrasign_v3.py --key None --list bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.0983 ] Assuming zero filled SBK key
[   0.1000 ] Sha saved in br_bct_BR.sha
[   0.1002 ] Warning: pub_key.key is not found
[   0.0996 ] Updating BCT with signature
[   0.0999 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesig bct_list_signed.xml
[   0.1001 ] Offset :4608 Len :3584
[   0.1005 ] Generating SHA2 Hash
[   0.1011 ] tegrasign_v3.py --key None --list bct_list.xml --sha sha512
[   0.1012 ] Assuming zero filled SBK key
[   0.1012 ] Assuming zero filled SBK key
[   0.1030 ] Sha saved in br_bct_BR.sha
[   0.1025 ] Updating BCT with SHA2 Hash
[   0.1028 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesha bct_list_signed.xml
[   0.1031 ] Offset :4608 Len :3584
[   0.1033 ] Offset :68 Len :8124
[   0.1035 ] Generating coldboot mb1-bct
[   0.1038 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct.cfg --misc tegra234-mb1-bct-misc-p3767-0000_cpp.dtb --device tegra234-mb1-bct-device-p3767-0000_cpp.dtb
[   0.1041 ] MB1-BCT version: 0.13

[   0.1057 ] Parsing config file :tegra234-mb1-bct-device-p3767-0000_cpp.dtb 
[   0.1059 ] Added Platform Config 9 data with size :- 100
[   0.1059 ] 
[   0.1059 ] Updating mb1-bct with firmware information
[   0.1062 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct_MB1.bct --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.1073 ] tegrahost_v2 --chip 0x23 0 --align mb1_cold_boot_bct_MB1_aligned.bct
[   0.1076 ] Generating SHA2 Hash for mb1bct
[   0.1091 ] Sha saved in mb1_cold_boot_bct_MB1_aligned.sha
[   0.1099 ] Sha saved in mb1_cold_boot_bct_MB1.sha
[   0.1096 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_cold_boot_bct_MB1_aligned.bct zerosbk
[   0.1099 ] adding BCH for mb1_cold_boot_bct_MB1_aligned.bct
[   0.1112 ] tegrasign_v3.py --key None --list mb1_cold_boot_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1113 ] Assuming zero filled SBK key
[   0.1124 ] Warning: pub_key.key is not found
[   0.1121 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_cold_boot_bct_MB1_aligned_sigheader.bct.encrypt mb1_cold_boot_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.1129 ] Generating recovery mb1-bct
[   0.1132 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct.cfg --misc tegra234-mb1-bct-misc-p3767-0000_cpp.dtb --device tegra234-mb1-bct-device-p3767-0000_cpp.dtb
[   0.1134 ] MB1-BCT version: 0.13

[   0.1150 ] Parsing config file :tegra234-mb1-bct-device-p3767-0000_cpp.dtb 
[   0.1152 ] Added Platform Config 9 data with size :- 100
[   0.1152 ] 
[   0.1152 ] Updating mb1-bct with firmware information
[   0.1155 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct_MB1.bct --recov --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.1165 ] tegrahost_v2 --chip 0x23 0 --align mb1_bct_MB1_aligned.bct
[   0.1169 ] Generating SHA2 Hash for mb1bct
[   0.1183 ] Sha saved in mb1_bct_MB1_aligned.sha
[   0.1191 ] Sha saved in mb1_bct_MB1.sha
[   0.1189 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_bct_MB1_aligned.bct zerosbk
[   0.1191 ] adding BCH for mb1_bct_MB1_aligned.bct
[   0.1204 ] tegrasign_v3.py --key None --list mb1_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1204 ] Assuming zero filled SBK key
[   0.1215 ] Warning: pub_key.key is not found
[   0.1212 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_bct_MB1_aligned_sigheader.bct.encrypt mb1_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.1220 ] Info: Skip generating mem_bct because sdram_config is not defined
[   0.1220 ] Info: Skip generating mem_bct because sdram_config is not defined
[   0.1220 ] Copying signatures
[   0.1224 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.1268 ] mb1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.1269 ] psc_bl1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.1269 ] Boot Rom communication
[   0.1272 ] tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[   0.1275 ] BR_CID: 0x80012344705DD3570800000001028140
[   0.1283 ] Sending bct_br
[   0.1284 ] Sending mb1
[   0.1288 ] Sending psc_bl1
[   0.1392 ] Sending bct_mb1
[   0.1449 ] Boot Rom communication completed
[   0.1456 ] tegrahost_v2 --chip 0x23 0 --align applet_t234_aligned.bin
[   0.1464 ] tegrahost_v2 --chip 0x23 0 --magicid MB2A --appendsigheader applet_t234_aligned.bin zerosbk
[   0.1467 ] adding BCH for applet_t234_aligned.bin
[   0.1558 ] tegrasign_v3.py --key None --list applet_t234_aligned_sigheader.bin_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1559 ] Assuming zero filled SBK key
[   0.1575 ] Warning: pub_key.key is not found
[   0.1573 ] tegrahost_v2 --chip 0x23 0 --updatesigheader applet_t234_aligned_sigheader.bin.encrypt applet_t234_aligned_sigheader.bin.hash zerosbk
[   0.1586 ] Sending mb2_applet...

[   0.1589 ] tegrarcm_v2 --chip 0x23 0 --pollbl --download applet applet_t234_sigheader.bin.encrypt
[   0.1592 ] BL: version 1.4.0.4-t234-54845784-e89ea9bc last_boot_error: 656573201
[   0.3689 ] Sending applet
[   0.3693 ] ERROR: might be timeout in USB write.
Error: Return value 3
Command tegrarcm_v2 --chip 0x23 0 --pollbl --download applet applet_t234_sigheader.bin.encrypt
[   0.4651 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.4665 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.4674 ] Retrieving board information
[   0.4679 ] tegrarcm_v2 --chip 0x23 0 --oem platformdetails chip chip_info.bin
[   0.4685 ] Retrieving EEPROM data
[   0.4685 ] tegrarcm_v2 --oem platformdetails eeprom cvm /home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/bootloader/cvm.bin --chip 0x23 0
[   0.4694 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.4703 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.4707 ] Dumping customer Info
[   0.4711 ] tegrarcm_v2 --chip 0x23 0 --oem dump bct tmp.bct
[   0.4719 ] tegrabct_v2 --brbct tmp.bct --chip 0x23 0 --custinfo /home/pkauls/jetpack_r36.4_2/Linux_for_Tegra/bootloader/custinfo_out.bin
[   0.4722 ] File tmp.bct open failed
[   0.4724 ] Error: try getting custinfo fail. Moving on
[   0.4724 ] Rebooting to recovery mode
[   0.4728 ] tegrarcm_v2 --chip 0x23 0 --ismb2
[   0.4732 ] Rebooting to recovery mode
[   0.4735 ] tegrarcm_v2 --chip 0x23 0 --reboot recovery
Parsing board information failed.
Error: failed to generate images
Cleaning up...
Flash Device UART Log 
jetson@jetson-dk-7:~$ sudo reboot forced-recovery
[sudo] password for jetson:
[  152.687146] tegra-ivc-bus bc00000.rtcpu:ivc-bus:echo@0: ivc channel driver missing
[  152.687158] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@1: ivc channel driver missing
[  152.687164] tegra-ivc-bus bc00000.rtcpu:ivc-bus:dbg@2: ivc channel driver missing
[  152.687171] tegra-ivc-bus bc00000.rtcpu:ivc-bus:diag@5: ivc channel driver missing
[  152.895796] nvgpu: 17000000.gpu             pmu_wait_message_cond:664  [ERR]  PMU wait timeout expired.
[  152.895803] nvgpu: 17000000.gpu             pmu_wait_message_cond:664  [ERR]  PMU wait timeout expired.
[  152.896467] nvgpu: 17000000.gpu      ga10b_intr_log_pending_intrs:306  [ERR]  Pending TOP[0]: 0x00000004, LEAF[4]: 0x11000000
[  152.897897] arm-smmu 12000000.iommu: disabling translation
[  152.898033] arm-smmu 10000000.iommu: disabling translation
[  152.898091] arm-smmu 8000000.iommu: disabling translation
[  152.898315] reboot: Restarting system with command 'forced-recovery'
▒▒▒▒Shutdown state requested 1
Rebooting system ...
▒▒
[0014.940] I> MB1 (version: 1.4.0.4-t234-54845784-e89ea9bc)
[0014.945] I> t234-A01-0-Silicon (0x12347) Prod
[0014.949] I> Boot-mode : BPMP Diagnostic
[0014.953] I> Entry timestamp: 0x00000000
[0014.957] I> last_boot_error: 0x0
[0014.960] I> BR-BCT: preprod_dev_sign: 0
[0014.964] I> rst_source: 0xb, rst_level: 0x1
[0014.968] I> Task: Initialize MB2 params
[0014.972] I> MB2-params @ 0x40060000
[0014.976] I> Task: Crypto init
[0014.979] I> Task: Perform MB1 KAT tests
[0014.983] I> Task: NVRNG health check
[0014.986] I> NVRNG: Health check success
[0014.990] I> Task: Secure debug controls
[0014.994] I> Task: strap war set
[0014.997] I> Task: Program NV master stream id
[0015.001] I> Task: Verify boot mode
[0015.007] I> Task: Alias fuses
[0015.010] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0015.017] I> Task: Print SKU type
[0015.020] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x000001c8
[0015.025] I> FUSE_OPT_GPC_DISABLE = 0x00000002
[0015.030] I> FUSE_OPT_TPC_DISABLE = 0x000000f0
[0015.034] I> FUSE_OPT_DLA_DISABLE = 0x00000003
[0015.038] I> FUSE_OPT_PVA_DISABLE = 0x00000001
[0015.042] I> FUSE_OPT_NVENC_DISABLE = 0x00000001
[0015.047] I> FUSE_OPT_NVDEC_DISABLE = 0x00000000
[0015.051] I> FUSE_OPT_FSI_DISABLE = 0x00000001
[0015.056] I> FUSE_OPT_EMC_DISABLE = 0x0000000c
[0015.060] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0015.064] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0015.068] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0015.072] I> FUSE_SKU_INFO_0 = 0xd5
[0015.076] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS
[0015.080] I> FUSE_PACKAGE_INFO_0 = 0x2
[0015.083] I> SKU: Prod
[0015.086] I> Task: Boost clocks
[0015.089] I> Initializing PLLC2 for AXI_CBB.
[0015.093] I> AXI_CBB : src = 35, divisor = 0
[0015.097] I> Task: Voltage monitor
[0015.100] I> VMON: Vmon re-calibration and fine tuning done
[0015.106] I> Task: UPHY init
[0015.109] W> UPHY: UPHY lane info table is empty in MB1 BCT.
[0015.115] I> Task: Boot device init
[0015.118] I> Boot_device: RCM
[0015.121] I> USB configuration success
[0015.125] I> Task: TSC init
[0015.128] I> Task: Enable WDT 5th expiry
[0015.131] I> Task: I2C register
[0015.134] I> Task: Set I2C bus freq
[0015.138] I> Task: Reset FSI
[0015.140] I> Task: Enable clock-mon
[0015.144] I> FMON: Fmon re-programming done
[0015.149] I> Task: Mapper init
[0015.151] I> Task: MB1 fixed firewalls
[0015.158] W> Firewall readback mismatch
[0015.163] I> Task: Load MB2/Applet/FSKP
[0015.167] I> Loading MB2 Applet
[0015.170] I> Slot: 1
[0015.172] E> BLOCK_DEV: Failed to open blockdev.
[0015.177] E> LOADER: Failed to open blockdev 0(0).
[0015.182] E> LOADER: Failed to get storage info for binary 21 from loader.
[0015.189] C> LOADER: Could not read binary 21.
[0015.194] E> Failed to load MB2
[0015.197] C> Task 0x46 failed (err: 0x27228311)
[0015.201] E> Top caller module: MB2_PARAMS, error module: LOADER, reason: 0x11, aux_info: 0x83
[0015.210] C> Boot Info Table status dump :
0111100000111000110111111111000000011110000000000000011000001
▒▒
18

okay, please let me check this with internal.
It may take some time for us to do further check and investigate for the root cause.

Before we found the root cause, you can just short REC/GND to enter into force recovery state before flash as WAR to prevent this issue.