We are using a custom carrier board for the AGX Orin with release 36.4.4.
I am following ‘Workflow 3: To massflash the backup image’ in tools/backup_restore/README_backup_restore.txt to perform system backup and restore operations. However, the restore process is unstable. It occasionally reports errors and fails to complete. The error log is as follows:
Tool log:
$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --massflash 1 --network usb0
# Entry added by NVIDIA initrd flash tool
/home/xx/code/nvidia/xx/v11/images/mfi_xx/tools/kernel_flash/tmp 127.0.0.1(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
rpcbind: another rpcbind is already running. Aborting
Export list for localhost:
/home/xx/code/nvidia/xx/v11/images/mfi_xx/tools/kernel_flash/tmp 127.0.0.1
10:45:11.265 - Debug: Checking UFW status and NFS port rules...
10:45:11.327 - Debug: UFW is not active. NFS port is accessible.
10:45:11.328 - Debug: Checking VPN connections...
10:45:11.334 - Debug: IPSec service is not running.
10:45:11.336 - Debug: VPN routes none in tun|tap|ppp|vpn|gpd.
10:45:11.338 - Debug: No VPN connection detected.
# Entry added by NVIDIA initrd flash tool
/home/xx/code/nvidia/xx/v11/images/mfi_xx/tools/kernel_flash/tmp 127.0.0.1(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
rpcbind: another rpcbind is already running. Aborting
Export list for localhost:
/home/xx/code/nvidia/xx/v11/images/mfi_xx/tools/kernel_flash/tmp 127.0.0.1
/home/xx/code/nvidia/xx/v11/images/mfi_xx/tools/kernel_flash/l4t_initrd_flash_internal.sh --network usb0 --usb-instance 1-6 --device-instance 0 --flash-only --network usb0 xxa internal
**********************************************
* *
* Step 1: Build the flashing environment *
* *
**********************************************
Create flash environment 0
/home/xx/code/nvidia/xx/v11/images/mfi_xx/bootloader /home/xx/code/nvidia/xx/v11/images/mfi_xx
/home/xx/code/nvidia/xx/v11/images/mfi_xx
Finish creating flash environment 0.
****************************************************
* *
* Step 2: Boot the device with flash initrd image *
* *
****************************************************
/home/xx/code/nvidia/xx/v11/images/mfi_xx/temp_initrdflash/bootloader0 /home/xx/code/nvidia/xx/v11/images/mfi_xx
./tegraflash.py --bl uefi_jetson_minimal_with_dtb_sigheader.bin.encrypt --bct br_bct_BR.bct --securedev --bldtb tegra234-xxa+p3701-0005-nv.dtb --applet rcm_2_encrypt.rcm --applet_softfuse rcm_1_encrypt.rcm --instance 1-6 --cmd "rcmboot" --cfg secureflash.xml --chip 0x23 --mb1_bct mb1_bct_MB1_sigheader.bct.encrypt --mem_bct mem_rcm_sigheader.bct.encrypt --mb1_cold_boot_bct mb1_cold_boot_bct_MB1_sigheader.bct.encrypt --mb1_bin mb1_t234_prod_aligned_sigheader.bin.encrypt --psc_bl1_bin psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --mem_bct_cold_boot mem_coldboot_sigheader.bct.encrypt --bins "psc_fw pscfw_t234_prod_sigheader.bin.encrypt; mts_mce mce_flash_o10_cr_prod_sigheader.bin.encrypt; tsec_fw tsec_t234_sigheader.bin.encrypt; mb2_applet applet_t234_sigheader.bin.encrypt; mb2_bootloader mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt; xusb_fw xusb_t234_prod_sigheader.bin.encrypt; pva_fw nvpva_020_sigheader.fw.encrypt; dce_fw display-t234-dce_sigheader.bin.encrypt; nvdec nvdec_t234_prod_sigheader.fw.encrypt; bpmp_fw bpmp_t234-TE990M-A1_prod_sigheader.bin.encrypt; bpmp_fw_dtb tegra234-bpmp-3701-0005-3737-0000_with_odm_sigheader.dtb.encrypt; rce_fw camera-rtcpu-t234-rce_sigheader.img.encrypt; ape_fw adsp-fw_sigheader.bin.encrypt; spe_fw spe_t234_sigheader.bin.encrypt; tos tos-optee_t234_sigheader.img.encrypt; eks eks_t234_sigheader.img.encrypt; kernel boot0.img; kernel_dtb tegra234-xxa+p3701-0005-nv.dtb" --bct_backup
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
Entering RCM boot
[ 0.0173 ] mb1_t234_prod_aligned_sigheader.bin.encrypt filename is from --mb1_bin
[ 0.0173 ] psc_bl1_t234_prod_aligned_sigheader.bin.encrypt filename is from --psc_bl1_bin
[ 0.0173 ] rcm boot with presigned binaries
[ 0.0179 ] tegrarcm_v2 --instance 1-6 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[ 0.0184 ] BR_CID: 0x80012344705DF8A33000000009010280
[ 0.0282 ] Sending bct_br
[ 0.0365 ] Sending mb1
[ 0.0371 ] Sending psc_bl1
[ 0.0497 ] Sending bct_mb1
[ 0.0562 ] Generating blob for T23x
[ 0.0582 ] tegrahost_v2 --chip 0x23 0 --generateblob blob.xml blob.bin
[ 0.0589 ] The number of images in blob is 19
[ 0.0600 ] blobsize is 121844455
[ 0.0602 ] Added binary blob_uefi_jetson_minimal_with_dtb_sigheader.bin.encrypt of size 2093120
[ 0.1071 ] Added binary blob_pscfw_t234_prod_sigheader.bin.encrypt of size 310768
[ 0.1073 ] Added binary blob_mce_flash_o10_cr_prod_sigheader.bin.encrypt of size 187120
[ 0.1074 ] Added binary blob_tsec_t234_sigheader.bin.encrypt of size 176128
[ 0.1075 ] Added binary blob_applet_t234_sigheader.bin.encrypt of size 279808
[ 0.1076 ] Not supported type: mb2_applet
[ 0.1076 ] Added binary blob_mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt of size 440944
[ 0.1078 ] Added binary blob_xusb_t234_prod_sigheader.bin.encrypt of size 164864
[ 0.1079 ] Added binary blob_nvpva_020_sigheader.fw.encrypt of size 2164640
[ 0.1086 ] Added binary blob_display-t234-dce_sigheader.bin.encrypt of size 12079200
[ 0.1137 ] Added binary blob_nvdec_t234_prod_sigheader.fw.encrypt of size 294912
[ 0.1150 ] Added binary blob_bpmp_t234-TE990M-A1_prod_sigheader.bin.encrypt of size 1027008
[ 0.1155 ] Added binary blob_tegra234-bpmp-3701-0005-3737-0000_with_odm_sigheader.dtb.encrypt of size 264960
[ 0.1156 ] Added binary blob_camera-rtcpu-t234-rce_sigheader.img.encrypt of size 458096
[ 0.1158 ] Added binary blob_adsp-fw_sigheader.bin.encrypt of size 415008
[ 0.1159 ] Added binary blob_spe_t234_sigheader.bin.encrypt of size 270336
[ 0.1160 ] Added binary blob_tos-optee_t234_sigheader.img.encrypt of size 1887312
[ 0.1168 ] Added binary blob_eks_t234_sigheader.img.encrypt of size 9232
[ 0.1170 ] Added binary blob_boot0.img of size 99061760
[ 0.1590 ] Added binary blob_tegra234-xxa+p3701-0005-nv.dtb of size 258135
[ 0.2143 ] tegrarcm_v2 --instance 1-6 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
[ 0.2147 ] BL: version 1.4.0.4-t234-54845784-e89ea9bc last_boot_error: 0
[ 0.3262 ] Sending bct_mem
[ 0.3396 ] Sending blob
[ 4.9401 ] RCM-boot started
/home/xx/code/nvidia/xxa/v11/images/mfi_xxa-v11-64gb-251124
***************************************
* *
* Step 3: Start the flashing process *
* *
***************************************
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for device to expose ssh ......Waiting for device to expose ssh ...Run command: flash on fc00:1:1:0::2
10:45:45.363 - Debug: Debug log saved to /tmp/tmp.we5PfdkdiT.
SSH ready
4194304
00:00:23.962 - Info: Serial Number: 1422125066920
00:00:24.008 - Info: Starting to create the GPT for the eMMC.
00:00:24.028 - Debug: Active index file is /mnt/internal/flash.idx
00:00:24.058 - Debug: Number of lines is 18
00:00:24.095 - Debug: max_index=17
00:00:24.505 - Debug: writing item=1, 1:3:primary_gpt,0,20480,mmcblk0_gptmbr.img,20480,fixed-<reserved>-0,eedfa609d108a44b8df2a000cb4ab8494ded48e3
00:00:51.542 - Debug: Writing primary_gpt partition with mmcblk0_gptmbr.img
00:00:52.264 - Debug: 20480 bytes from /mnt/internal/mmcblk0_gptmbr.img to /dev/mmcblk0: 1KB block=20 remainder=0
00:00:52.269 - Debug: dd if=/mnt/internal/mmcblk0_gptmbr.img of=/dev/mmcblk0 bs=1K skip=0 seek=0 count=20
20+0 records in
20+0 records out
20480 bytes (20 kB, 20 KiB) copied, 0.0033471 s, 6.1 MB/s
00:00:52.312 - Debug: Writing primary_gpt partition done
00:00:52.578 - Debug: Writing secondary_gpt partition with mmcblk0_gptbackup.img
00:00:53.653 - Debug: Offset is not aligned to K Bytes, no optimization is applied
00:00:53.657 - Debug: dd if=/mnt/internal/mmcblk0_gptbackup.img of=/dev/mmcblk0 bs=1 skip=0 seek=63585631744 count=16896
16896+0 records in
16896+0 records out
16896 bytes (17 kB, 16 KiB) copied, 0.0229961 s, 735 kB/s
00:00:53.686 - Debug: Writing secondary_gpt partition done
00:00:54.105 - Info: Successfully created the GPT for the eMMC.
00:00:54.125 - Info: Starting to flash the eMMC.
00:00:54.126 - Debug: Flash index file is /mnt/internal/flash.idx
00:00:54.129 - Debug: Active index file is /mnt/internal/flash.idx
00:00:54.130 - Debug: Number of lines is 18
00:00:54.134 - Debug: max_index=17
00:00:54.146 - Info: Starting to flash the QSPI.
QSPI storage size: 67108864 bytes.
00:00:54.133 - Debug: Number of lines is 18
00:00:54.175 - Debug: max_index=17
00:00:54.200 - Debug: writing item=1, 1:3:primary_gpt,0,20480,mmcblk0_gptmbr.img,20480,fixed-<reserved>-0,eedfa609d108a44b8df2a000cb4ab8494ded48e3
00:00:54.214 - Debug: writing item=2, 1:3:secondary_gpt,63585631744,16896,mmcblk0_gptbackup.img,16896,fixed-<reserved>-0,8637d8cff9753cf579688962d29059af09ff94ca
00:00:54.227 - Debug: writing item=3, 1:3:APP,0,62024003584,mmcblk0p1_bak.img,62024003584,fixed-<reserved>-1,c631908efd3ea1d1aa9beb8df200246023b5e3ba
00:00:54.966 - Debug: Formatting APP partition /dev/mmcblk0p1 ...
mke2fs 1.46.5 (30-Dec-2021)
Discarding device blocks: done
Creating filesystem with 15142579 4k blocks and 3792896 inodes
Filesystem UUID: 0f2330c1-eb6b-461f-8dbb-6638d671f11f
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424
Allocating group tables: done
Writing inode tables: done
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done
00:01:06.164 - Debug: Formatting APP parition done
00:01:06.289 - Debug: Formatting APP partition /dev/mmcblk0p1 ...
" -x -I 'zstd -T0' -pf /mnt/internal/mmcblk0p1_bak.img --warning=no-timestamp --numeric-owner --xattrs --xattrs-include=* -C /tmp/ci-DY8X7mVKFk
MEMERASE: Input/output error
Uart log:
[ 54.204151] 00:00:54.200 - Debug: writing item=1, 1:3:primary_gpt,0,20480,mmcblk0_gptmbr.img,20480,fixed-<reserved>-0,eedfa609d108a44b8df2a000cb4ab8494ded48e3
[ 54.204151]
[ 54.217528] 00:00:54.214 - Debug: writing item=2, 1:3:secondary_gpt,63585631744,16896,mmcblk0_gptbackup.img,16896,fixed-<reserved>-0,8637d8cff9753cf579688962d29059af09ff94ca
[ 54.217528]
[ 54.230812] 00:00:54.227 - Debug: writing item=3, 1:3:APP,0,62024003584,mmcblk0p1_bak.img,62024003584,fixed-<reserved>-1,c631908efd3ea1d1aa9beb8df200246023b5e3ba
[ 54.230812]
[ 54.969882] 00:00:54.966 - Debug: Formatting APP partition /dev/mmcblk0p1 ...
bash-5.1# [ 66.168946] 00:01:06.164 - Debug: Formatting APP parition done
[ 66.287786] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 66.293069] 00:01:06.289 - Debug: Formatting APP partition /dev/mmcblk0p1 ...
" -x -I 'zstd -T0' -pf /mnt/internal/mmcblk0p1_bak.img --warning=no-timestamp --numeric-owner --xattrs --xattrs-include=* -C /tmp/ci-DY8X7mVKFk
[ 117.917581] ------------[ cut here ]------------
[ 117.917589] WARNING: CPU: 2 PID: 876 at /home/xx/code/nvidia/xxa/v11/Linux_for_Tegra/source/kernel_out/nvidia-oot/drivers/spi/spi-tegra210-quad.c:1406 tegra_qspi_transfer_one_message+0x36c/0x7b4 [spi_tegra210_quad]
[ 117.917610] Modules linked in: usb_f_rndis u_ether spi_tegra210_quad(O) tegra_mce(O) stusb160x nvethernet(O) nvpps(O) ipv6 tegra_xudc ucsi_ccg typec_ucsi typec libcomposite pwm_fan pwm_tegra tegra_bpmp_thermal nvme nvme_core pcie_tegra194 phy_tegra194_p2u r8168(O)
[ 117.917633] CPU: 2 PID: 876 Comm: mtd_debug Tainted: G O 5.15.148-tegra #1
[ 117.917636] Hardware name: NVIDIA NAMTSO xxA/Jetson, BIOS 36.4.4-gcid-41062509 06/16/2025
[ 117.917638] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 117.917641] pc : tegra_qspi_transfer_one_message+0x36c/0x7b4 [spi_tegra210_quad]
[ 117.917644] lr : tegra_qspi_transfer_one_message+0x2d0/0x7b4 [spi_tegra210_quad]
[ 117.917646] sp : ffff80000e1535a0
[ 117.917647] x29: ffff80000e1535a0 x28: ffff00008f316c80 x27: ffff000f695d26d0
[ 117.917649] x26: ffff80000e153848 x25: 0000000000000000 x24: ffff000f695d2600
[ 117.917651] x23: ffff000f695d2000 x22: ffff000f695d3800 x21: 0000000000000000
[ 117.917653] x20: ffff80000e153750 x19: ffff80000e153848 x18: 0000000000000000
[ 117.917655] x17: 000000040044ffff x16: 005000f5b5503510 x15: 0000000000000000
[ 117.917658] x14: ffff0000801dcd80 x13: ffff56a6f20a9000 x12: 00000000b464d91d
[ 117.917660] x11: 0000000000000000 x10: ab6bfff3632ce41a x9 : f4397b68d8270096
[ 117.917662] x8 : ffff00008f317bd8 x7 : 0000000000000003 x6 : 00000000eece0f29
[ 117.917664] x5 : 00000000410fd420 x4 : 0000000000c0000e x3 : 0000000000000000
[ 117.917667] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
[ 117.917670] Call trace:
[ 117.917672] tegra_qspi_transfer_one_message+0x36c/0x7b4 [spi_tegra210_quad]
[ 117.917674] __spi_pump_messages+0x384/0x7f0
[ 117.917689] __spi_sync+0x2d0/0x310
[ 117.917693] spi_sync+0x3c/0x60
[ 117.917695] spi_mem_exec_op+0x30c/0x440
[ 117.917698] spi_nor_read_sr+0xa8/0x140
[ 117.917703] spi_nor_wait_till_ready_with_timeout+0xc0/0x460
[ 117.917706] spi_nor_erase+0x218/0x640
[ 117.917707] mtd_erase+0x118/0x1d0
[ 117.917712] mtdchar_ioctl+0xf08/0x1000
[ 117.917715] mtdchar_unlocked_ioctl+0x60/0xe0
[ 117.917718] __arm64_sys_ioctl+0xb4/0x100
[ 117.917722] invoke_syscall+0x5c/0x130
[ 117.917727] el0_svc_common.constprop.0+0x64/0x110
[ 117.917730] do_el0_svc+0x74/0xa0
[ 117.917732] el0_svc+0x28/0x80
[ 117.917737] el0t_64_sync_handler+0xa4/0x130
[ 117.917738] el0t_64_sync+0x1a4/0x1a8
[ 117.917741] ---[ end trace 612eb88f23658d66 ]---
[ 118.922659] tegra-qspi 3270000.spi: transfer timeout
[ 118.922661] tegra-qspi 3270000.spi: error in transfer, fifo status 0x00c00004
[ 118.923057] spi_master spi0: failed to transfer one message from queue
This issue occurs intermittently (it is probabilistic). How can I resolve this?
