Some questions About Bootloader Redundancy

It can be accessed through serial console from host no matter you are using x86 or ARM based PC.

They can only be accessed in recovery boot.

nvbootctrl can do this work. You can create a custom service to run this command and detect or notify the current status.

Hi,KevinFFF

Okay, I can consider adding a system service later to monitor this. If it switches to Slot B, the service can automatically restore Slot A’s bootloader using nv_bootloader_capsule_updater.sh.

When you flash the external device, it will enter into recovery kernel to perform the flash process.
You can simply enter into this state through UEFI menu: ESC → Device Manager → NVIDIA Configuration → L4T Configuration → L4T Boot Mode → Recovery Partition → Reset
You would need a host PC and USB-C and serial console connected on your board for this task.

In serial console, you should be able to see:

bash-5.2#

Hi,KevinFFF

Sorry, I couldn’t find the MTD-related nodes.

bash-5.1# ls mtd
ls: cannot access 'mtd': No such file or directory
bash-5.1# ls
autofs              mmcblk0boot1  nvme0n1p3  teepriv0  tty38    ttyS1
bus                 mmcblk0p1     nvme0n1p4  tty       tty39    ttyS2
console             mmcblk0p10    nvme0n1p5  tty0      tty4     ttyS3
cpu_dma_latency     mmcblk0p11    nvme0n1p6  tty1      tty40    ttyTCU0
efi_capsule_loader  mmcblk0p12    nvme0n1p7  tty10     tty41    ttyTHS1
full                mmcblk0p13    nvme0n1p8  tty11     tty42    ttyTHS2
gpiochip0           mmcblk0p14    nvme0n1p9  tty12     tty43    ttyp0
gpiochip1           mmcblk0p15    port       tty13     tty44    ttyp1
hwrng               mmcblk0p2     pps0       tty14     tty45    ttyp2
i2c-0               mmcblk0p3     ptmx       tty15     tty46    ttyp3
i2c-1               mmcblk0p4     ptyp0      tty16     tty47    ttyp4
i2c-2               mmcblk0p5     ptyp1      tty17     tty48    ttyp5
i2c-3               mmcblk0p6     ptyp2      tty18     tty49    ttyp6
i2c-4               mmcblk0p7     ptyp3      tty19     tty5     ttyp7
i2c-5               mmcblk0p8     ptyp4      tty2      tty50    ttyp8
i2c-6               mmcblk0p9     ptyp5      tty20     tty51    ttyp9
i2c-7               mmcblk0rpmb   ptyp6      tty21     tty52    ttypa
i2c-8               net           ptyp7      tty22     tty53    ttypb
input               ng0n1         ptyp8      tty23     tty54    ttypc
kmsg                null          ptyp9      tty24     tty55    ttypd
kvm                 nvme-fabrics  ptypa      tty25     tty56    ttype
loop-control        nvme0         ptypb      tty26     tty57    ttypf
loop0               nvme0n1       ptypc      tty27     tty58    urandom
loop1               nvme0n1p1     ptypd      tty28     tty59    vcs
loop2               nvme0n1p10    ptype      tty29     tty6     vcs1
loop3               nvme0n1p11    ptypf      tty3      tty60    vcsa
loop4               nvme0n1p12    random     tty30     tty61    vcsa1
loop5               nvme0n1p13    rtc1       tty31     tty62    vcsu
loop6               nvme0n1p14    snd        tty32     tty63    vcsu1
loop7               nvme0n1p15    spidev0.0  tty33     tty7     vfio
mapper              nvme0n1p16    spidev0.1  tty34     tty8     vga_arbiter
mem                 nvme0n1p17    spidev1.0  tty35     tty9     watchdog
mmcblk0             nvme0n1p18    spidev1.1  tty36     ttyAMA0  watchdog0
mmcblk0boot0        nvme0n1p2     tee0       tty37     ttyS0    zero
bash-5.1# cat /proc/mtd
dev:    size   erasesize  name
bash-5.1#

Is it mmcblk0boot0 and mmcblk0boot1?

Hi,KevinFFF

When both slots of the device are damaged, the device will enter recovery mode (recovery boot). Doesn’t the device automatically execute the OTA script, like nv_ota_run_tasks.sh? I noticed in the development documentation that OTA is mainly used for upgrading between different SDK versions. I’m wondering if it’s possible to create an offline image that packages my current rootfs and bootloader, using the same SDK version, and then automatically restore both slots once the device enters recovery mode? Could you explain this in more detail? If this is possible, how can I verify it? I noticed that the OTA development documentation doesn’t explain this situation in detail. For the scenario I want to implement, can this OTA process be done remotely on the device without relying on a host computer?"

If OTA can fulfill the requirements I mentioned above, how exactly do I create the offline OTA image package? Can it be placed on an NVMe drive or other storage media?After the device enters recovery mode, How is it associated with nv_ota_run_tasks.sh to automatically trigger the update? After the update, which slot will the device switch to?

They are the boot device of eMMC.

Sorry that I might provide the wrong method to enter recovery boot for your to access QSPI.

Please refer to the following steps instead.

  1. configure the board in force recovery state.
  2. connect USB cable to the host
  3. run the following command on host for RCM boot
$ sudo ./flash.sh --rcm-boot jetson-agx-orin-devkit mmcblk0p1
  1. you should see the following messages in serial console.
L4TLauncher: Attempting RCM Boot
  1. Run the following command to check.
$ ls /dev/mtd*
/dev/mtd0  /dev/mtd0ro  /dev/mtdblock0

Hi,KevinFFF

Thank you — I think I can now see the NOR flash device nodes. Although the serial log did not show L4TLauncher: Attempting RCM Boot, the host computer did indicate it entered RCM boot. I want to understand: with only three device nodes, how could I intentionally damage slot A or slot B?

root@tegra-ubuntu:/dev# ls
autofs              ng0n1                         rtc             tty55
block               null                          rtc0            tty56
btrfs-control       nvgpu                         rtc1            tty57
bus                 nvhost-as-gpu                 rtc2            tty58
char                nvhost-ctrl-gpu               shm             tty59
console             nvhost-ctrl-nvdla0            snd             tty6
cpu_dma_latency     nvhost-ctrl-nvdla1            spidev0.0       tty60
cuse                nvhost-ctrl-pva0              spidev0.1       tty61
disk                nvhost-ctxsw-gpu              spidev1.0       tty62
dri                 nvhost-dbg-gpu                spidev1.1       tty63
efi_capsule_loader  nvhost-gpu                    stderr          tty7
fd                  nvhost-nvsched-gpu            stdin           tty8
full                nvhost-nvsched_ctrl_fifo-gpu  stdout          tty9
fuse                nvhost-power-gpu              tee0            ttyAMA0
gpiochip0           nvhost-prof-ctx-gpu           teepriv0        ttyGS0
gpiochip1           nvhost-prof-dev-gpu           tegra-soc-hwpm  ttyS0
host1x-fence        nvhost-prof-gpu               tty             ttyS1
hugepages           nvhost-sched-gpu              tty0            ttyS2
hwrng               nvhost-tsg-gpu                tty1            ttyS3
i2c-0               nvidia0                       tty10           ttyTCU0
i2c-1               nvidiactl                     tty11           ttyTHS1
i2c-2               nvmap                         tty12           ttyTHS2
i2c-3               nvme-fabrics                  tty13           ttyp0
i2c-4               nvme0                         tty14           ttyp1
i2c-5               nvme0n1                       tty15           ttyp2
i2c-6               nvme0n1p1                     tty16           ttyp3
i2c-7               nvme0n1p10                    tty17           ttyp4
i2c-8               nvme0n1p11                    tty18           ttyp5
initctl             nvme0n1p12                    tty19           ttyp6
input               nvme0n1p13                    tty2            ttyp7
kmsg                nvme0n1p14                    tty20           ttyp8
kvm                 nvme0n1p15                    tty21           ttyp9
log                 nvme0n1p16                    tty22           ttypa
loop-control        nvme0n1p17                    tty23           ttypb
loop0               nvme0n1p18                    tty24           ttypc
loop1               nvme0n1p2                     tty25           ttypd
loop2               nvme0n1p3                     tty26           ttype
loop3               nvme0n1p4                     tty27           ttypf
loop4               nvme0n1p5                     tty28           urandom
loop5               nvme0n1p6                     tty29           v4l2-nvdec
loop6               nvme0n1p7                     tty3            v4l2-nvenc
loop7               nvme0n1p8                     tty30           vcs
mapper              nvme0n1p9                     tty31           vcs1
mem                 nvpps0                        tty32           vcs2
mmcblk0             nvsciipc                      tty33           vcs3
mmcblk0boot0        port                          tty34           vcs4
mmcblk0boot1        pps0                          tty35           vcs5
mmcblk0p1           ptmx                          tty36           vcs6
mmcblk0p10          ptp0                          tty37           vcsa
mmcblk0p11          pts                           tty38           vcsa1
mmcblk0p12          ptyp0                         tty39           vcsa2
mmcblk0p13          ptyp1                         tty4            vcsa3
mmcblk0p14          ptyp2                         tty40           vcsa4
mmcblk0p15          ptyp3                         tty41           vcsa5
mmcblk0p2           ptyp4                         tty42           vcsa6
mmcblk0p3           ptyp5                         tty43           vcsu
mmcblk0p4           ptyp6                         tty44           vcsu1
mmcblk0p5           ptyp7                         tty45           vcsu2
mmcblk0p6           ptyp8                         tty46           vcsu3
mmcblk0p7           ptyp9                         tty47           vcsu4
mmcblk0p8           ptypa                         tty48           vcsu5
mmcblk0p9           ptypb                         tty49           vcsu6
mmcblk0rpmb         ptypc                         tty5            vfio
mqueue              ptypd                         tty50           vga_arbiter
mtd0                ptype                         tty51           watchdog
mtd0ro              ptypf                         tty52           watchdog0
mtdblock0           random                        tty53           zero
net                 rfkill                        tty54
root@tegra-ubuntu:/dev# ping baidu.com

[   0.0484 ] Sending bct_br
[   0.0776 ] Sending mb1
[   0.0785 ] Sending psc_bl1
[   0.1013 ] Sending bct_mb1
[   0.1140 ] Generating blob for T23x
[   0.1163 ] tegrahost_v2 --chip 0x23 0 --generateblob blob.xml blob.bin
[   0.1178 ] The number of images in blob is 20
[   0.1185 ] blobsize is 84522284
[   0.1186 ] Added binary blob_uefi_jetson_with_dtb_sigheader.bin.encrypt of size 3584064
[   0.1669 ] Added binary blob_pscfw_t234_prod_sigheader.bin.encrypt of size 310768
[   0.1672 ] Added binary blob_mce_flash_o10_cr_prod_sigheader.bin.encrypt of size 187120
[   0.1676 ] Added binary blob_tsec_t234_sigheader.bin.encrypt of size 176128
[   0.1679 ] Added binary blob_applet_t234_sigheader.bin.encrypt of size 279616
[   0.1680 ] Not supported type: mb2_applet
[   0.1681 ] Added binary blob_mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt of size 439968
[   0.1683 ] Added binary blob_xusb_t234_prod_sigheader.bin.encrypt of size 164864
[   0.1685 ] Added binary blob_nvpva_020_sigheader.fw.encrypt of size 2164640
[   0.1686 ] Added binary blob_display-t234-dce_sigheader.bin.encrypt of size 12065600
[   0.1758 ] Added binary blob_nvdec_t234_prod_sigheader.fw.encrypt of size 294912
[   0.1790 ] Added binary blob_bpmp_t234-TE992M-A1_prod_sigheader.bin.encrypt of size 1046080
[   0.1792 ] Added binary blob_fsi-fw-ecc_sigheader.bin.encrypt of size 5746688
[   0.1851 ] Added binary blob_tegra234-bpmp-3701-0008-3737-0000_with_odm_sigheader.dtb.encrypt of size 156096
[   0.1892 ] Added binary blob_camera-rtcpu-t234-rce_sigheader.img.encrypt of size 458096
[   0.1894 ] Added binary blob_adsp-fw_sigheader.bin.encrypt of size 414976
[   0.1895 ] Added binary blob_spe_t234_sigheader.bin.encrypt of size 270336
[   0.1897 ] Added binary blob_tos-optee_t234_sigheader.img.encrypt of size 1600576
[   0.1899 ] Added binary blob_eks_t234_sigheader.img.encrypt of size 9232
[   0.1942 ] Added binary blob_boot.img of size 54906880
[   0.2313 ] Added binary blob_tegra234-p3737-0000+p3701-0008-nv.dtb of size 244540
[   0.3003 ] tegrarcm_v2 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
[   0.3008 ] BL: version 1.4.0.2-t234-54845784-08a4de08 last_boot_error: 0
[   0.3762 ] Sending bct_mem
[   0.4006 ] Sending blob
[   8.0134 ] RCM-boot started

Hi,KevinFFF

How about this question ? May I ask if OTA can solve the redundancy recovery issue independently, without relying on the Host computer?

You can simply remove /lib so that your board should not boot correctly.

To perform an OTA update, you must first boot the device so that you can place the OTA payload (the ota_payload_package.tar.gz file) onto it.
Regarding your assumption, using OTA to recover an unbootable slot is certainly available. For example, if Slot A is unbootable, you can successfully perform the OTA update from Slot B.

Hi,KevinFFF

Is this kind of OTA recovery automatic — for example, does Slot B automatically recover Slot A?
If both Slot A and Slot B are damaged, the device will enter recovery mode. In that case, will it also recover automatically?
Which slot will it recover?
And after recovery, which slot will the device boot from?
Please help clarify these questions, thank you.

It will not be recovered automatically.
You have to perform OTA update or Capsule update manually to recover it.

You have to recover it manually.
If it is caused by something removed in rootfs, then your can put that back to recover.

Please try to mount them and check what are included first.

Hi,KevinFFF

I’m very sorry,the filesystem cannot be mounted properly.

root@tegra-ubuntu:/mnt# mount /dev/mtd
mtd0       mtd0ro     mtdblock0
root@tegra-ubuntu:/mnt# mount /dev/mtd0 test/
mount: /mnt/test: /dev/mtd0 is not a block device.
root@tegra-ubuntu:/mnt# mount /dev/mtd0ro test/
mount: /mnt/test: /dev/mtd0ro is not a block device.
root@tegra-ubuntu:/mnt# mount /dev/mtdblock0 test/
mount: /mnt/test: wrong fs type, bad option, bad superblock on /dev/mtdblock0, missing codepage or helper program, or other error.
root@tegra-ubuntu:/mnt# umount test/
umount: test/: not mounted.
root@tegra-ubuntu:/mnt# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 04000000 00001000 "spi2.0"
root@tegra-ubuntu:/mnt# dmesg | grep mtd
root@tegra-ubuntu:/mnt#
root@tegra-ubuntu:/mnt#

The OTA process will run only if there’s the OTA payload exists and you have to put that payload manually.

May I know what’s your requirement now? To break some partition in QSPI to check if the redundant bootloader mechanism works

Hi,KevinFFF

Yes. I need a complete test procedure: corrupt Slot A, try booting three times and verify it falls back to Slot B; a script should be able to perform the upgrade and recovery, and upon successful recovery switch back to Slot A.

Please refer to <Linux_for_Tegra>/tools/ota_tools/version_upgrade/Image_based_OTA_Examples.txt for the detailed workflow to perform OTA update. There are many use cases and steps that you can refer to.
Before OTA update, you may need to run the following command to trigger it.

$ sudo ./nv_ota_start.sh ~/ota_payload_package.tar.gz

I would suggest you also enabling rootfs a/b and you can simply remove /lib in rootfs-A so that it will boot failed and boot into slot B after 3 times trials. It is the similar flow as I’ve verified in previous r35 release on T194 series.
Rootfs A/B redundancy fail-over mechanism in Jetpack5.1