It can be accessed through serial console from host no matter you are using x86 or ARM based PC.
They can only be accessed in recovery boot.
nvbootctrl can do this work. You can create a custom service to run this command and detect or notify the current status.
Hi,KevinFFF
KevinFFF:
It can be accessed through serial console from host no matter you are using x86 or ARM based PC.
Does the recovery boot here refer to the OTA after entering recovery mode? I would like to ask whether it’s possible to perform recovery remotely (including Bootloader and rootfs) without using any PC, and this is not referring to a JetPack version upgrade.
They can only be accessed in recovery boot.
Okay, I’ll force the device into recovery mode and give it a try.
nvbootctrl can do this work. You can create a custom service to run this command and detect or notify the current status.
Okay, I can consider adding a system service later to monitor this. If it switches to Slot B, the service can automatically restore Slot A’s bootloader using nv_bootloader_capsule_updater.sh.
When you flash the external device, it will enter into recovery kernel to perform the flash process.
You can simply enter into this state through UEFI menu: ESC → Device Manager → NVIDIA Configuration → L4T Configuration → L4T Boot Mode → Recovery Partition → Reset
You would need a host PC and USB-C and serial console connected on your board for this task.
In serial console, you should be able to see:
bash-5.2#
Hi,KevinFFF
KevinFFF:
When you flash the external device, it will enter into recovery kernel to perform the flash process.
You can simply enter into this state through UEFI menu: ESC → Device Manager → NVIDIA Configuration → L4T Configuration → L4T Boot Mode → Recovery Partition → Reset
You would need a host PC and USB-C and serial console connected on your board for this task.
In serial console, you should be able to see:
bash-5.2#
Sorry, I couldn’t find the MTD-related nodes.
bash-5.1# ls mtd
ls: cannot access 'mtd': No such file or directory
bash-5.1# ls
autofs mmcblk0boot1 nvme0n1p3 teepriv0 tty38 ttyS1
bus mmcblk0p1 nvme0n1p4 tty tty39 ttyS2
console mmcblk0p10 nvme0n1p5 tty0 tty4 ttyS3
cpu_dma_latency mmcblk0p11 nvme0n1p6 tty1 tty40 ttyTCU0
efi_capsule_loader mmcblk0p12 nvme0n1p7 tty10 tty41 ttyTHS1
full mmcblk0p13 nvme0n1p8 tty11 tty42 ttyTHS2
gpiochip0 mmcblk0p14 nvme0n1p9 tty12 tty43 ttyp0
gpiochip1 mmcblk0p15 port tty13 tty44 ttyp1
hwrng mmcblk0p2 pps0 tty14 tty45 ttyp2
i2c-0 mmcblk0p3 ptmx tty15 tty46 ttyp3
i2c-1 mmcblk0p4 ptyp0 tty16 tty47 ttyp4
i2c-2 mmcblk0p5 ptyp1 tty17 tty48 ttyp5
i2c-3 mmcblk0p6 ptyp2 tty18 tty49 ttyp6
i2c-4 mmcblk0p7 ptyp3 tty19 tty5 ttyp7
i2c-5 mmcblk0p8 ptyp4 tty2 tty50 ttyp8
i2c-6 mmcblk0p9 ptyp5 tty20 tty51 ttyp9
i2c-7 mmcblk0rpmb ptyp6 tty21 tty52 ttypa
i2c-8 net ptyp7 tty22 tty53 ttypb
input ng0n1 ptyp8 tty23 tty54 ttypc
kmsg null ptyp9 tty24 tty55 ttypd
kvm nvme-fabrics ptypa tty25 tty56 ttype
loop-control nvme0 ptypb tty26 tty57 ttypf
loop0 nvme0n1 ptypc tty27 tty58 urandom
loop1 nvme0n1p1 ptypd tty28 tty59 vcs
loop2 nvme0n1p10 ptype tty29 tty6 vcs1
loop3 nvme0n1p11 ptypf tty3 tty60 vcsa
loop4 nvme0n1p12 random tty30 tty61 vcsa1
loop5 nvme0n1p13 rtc1 tty31 tty62 vcsu
loop6 nvme0n1p14 snd tty32 tty63 vcsu1
loop7 nvme0n1p15 spidev0.0 tty33 tty7 vfio
mapper nvme0n1p16 spidev0.1 tty34 tty8 vga_arbiter
mem nvme0n1p17 spidev1.0 tty35 tty9 watchdog
mmcblk0 nvme0n1p18 spidev1.1 tty36 ttyAMA0 watchdog0
mmcblk0boot0 nvme0n1p2 tee0 tty37 ttyS0 zero
bash-5.1# cat /proc/mtd
dev: size erasesize name
bash-5.1#
Is it mmcblk0boot0 and mmcblk0boot1?
Hi,KevinFFF
When both slots of the device are damaged, the device will enter recovery mode (recovery boot). Doesn’t the device automatically execute the OTA script, like nv_ota_run_tasks.sh? I noticed in the development documentation that OTA is mainly used for upgrading between different SDK versions. I’m wondering if it’s possible to create an offline image that packages my current rootfs and bootloader, using the same SDK version, and then automatically restore both slots once the device enters recovery mode? Could you explain this in more detail? If this is possible, how can I verify it? I noticed that the OTA development documentation doesn’t explain this situation in detail. For the scenario I want to implement, can this OTA process be done remotely on the device without relying on a host computer?"
If OTA can fulfill the requirements I mentioned above, how exactly do I create the offline OTA image package? Can it be placed on an NVMe drive or other storage media?After the device enters recovery mode, How is it associated with nv_ota_run_tasks.sh to automatically trigger the update? After the update, which slot will the device switch to?
They are the boot device of eMMC.
Sorry that I might provide the wrong method to enter recovery boot for your to access QSPI.
Please refer to the following steps instead.
configure the board in force recovery state.
connect USB cable to the host
run the following command on host for RCM boot
$ sudo ./flash.sh --rcm-boot jetson-agx-orin-devkit mmcblk0p1
you should see the following messages in serial console.
L4TLauncher: Attempting RCM Boot
Run the following command to check.
$ ls /dev/mtd*
/dev/mtd0 /dev/mtd0ro /dev/mtdblock0
Hi,KevinFFF
Thank you — I think I can now see the NOR flash device nodes. Although the serial log did not show L4TLauncher: Attempting RCM Boot, the host computer did indicate it entered RCM boot. I want to understand: with only three device nodes, how could I intentionally damage slot A or slot B?
root@tegra-ubuntu:/dev# ls
autofs ng0n1 rtc tty55
block null rtc0 tty56
btrfs-control nvgpu rtc1 tty57
bus nvhost-as-gpu rtc2 tty58
char nvhost-ctrl-gpu shm tty59
console nvhost-ctrl-nvdla0 snd tty6
cpu_dma_latency nvhost-ctrl-nvdla1 spidev0.0 tty60
cuse nvhost-ctrl-pva0 spidev0.1 tty61
disk nvhost-ctxsw-gpu spidev1.0 tty62
dri nvhost-dbg-gpu spidev1.1 tty63
efi_capsule_loader nvhost-gpu stderr tty7
fd nvhost-nvsched-gpu stdin tty8
full nvhost-nvsched_ctrl_fifo-gpu stdout tty9
fuse nvhost-power-gpu tee0 ttyAMA0
gpiochip0 nvhost-prof-ctx-gpu teepriv0 ttyGS0
gpiochip1 nvhost-prof-dev-gpu tegra-soc-hwpm ttyS0
host1x-fence nvhost-prof-gpu tty ttyS1
hugepages nvhost-sched-gpu tty0 ttyS2
hwrng nvhost-tsg-gpu tty1 ttyS3
i2c-0 nvidia0 tty10 ttyTCU0
i2c-1 nvidiactl tty11 ttyTHS1
i2c-2 nvmap tty12 ttyTHS2
i2c-3 nvme-fabrics tty13 ttyp0
i2c-4 nvme0 tty14 ttyp1
i2c-5 nvme0n1 tty15 ttyp2
i2c-6 nvme0n1p1 tty16 ttyp3
i2c-7 nvme0n1p10 tty17 ttyp4
i2c-8 nvme0n1p11 tty18 ttyp5
initctl nvme0n1p12 tty19 ttyp6
input nvme0n1p13 tty2 ttyp7
kmsg nvme0n1p14 tty20 ttyp8
kvm nvme0n1p15 tty21 ttyp9
log nvme0n1p16 tty22 ttypa
loop-control nvme0n1p17 tty23 ttypb
loop0 nvme0n1p18 tty24 ttypc
loop1 nvme0n1p2 tty25 ttypd
loop2 nvme0n1p3 tty26 ttype
loop3 nvme0n1p4 tty27 ttypf
loop4 nvme0n1p5 tty28 urandom
loop5 nvme0n1p6 tty29 v4l2-nvdec
loop6 nvme0n1p7 tty3 v4l2-nvenc
loop7 nvme0n1p8 tty30 vcs
mapper nvme0n1p9 tty31 vcs1
mem nvpps0 tty32 vcs2
mmcblk0 nvsciipc tty33 vcs3
mmcblk0boot0 port tty34 vcs4
mmcblk0boot1 pps0 tty35 vcs5
mmcblk0p1 ptmx tty36 vcs6
mmcblk0p10 ptp0 tty37 vcsa
mmcblk0p11 pts tty38 vcsa1
mmcblk0p12 ptyp0 tty39 vcsa2
mmcblk0p13 ptyp1 tty4 vcsa3
mmcblk0p14 ptyp2 tty40 vcsa4
mmcblk0p15 ptyp3 tty41 vcsa5
mmcblk0p2 ptyp4 tty42 vcsa6
mmcblk0p3 ptyp5 tty43 vcsu
mmcblk0p4 ptyp6 tty44 vcsu1
mmcblk0p5 ptyp7 tty45 vcsu2
mmcblk0p6 ptyp8 tty46 vcsu3
mmcblk0p7 ptyp9 tty47 vcsu4
mmcblk0p8 ptypa tty48 vcsu5
mmcblk0p9 ptypb tty49 vcsu6
mmcblk0rpmb ptypc tty5 vfio
mqueue ptypd tty50 vga_arbiter
mtd0 ptype tty51 watchdog
mtd0ro ptypf tty52 watchdog0
mtdblock0 random tty53 zero
net rfkill tty54
root@tegra-ubuntu:/dev# ping baidu.com
[ 0.0484 ] Sending bct_br
[ 0.0776 ] Sending mb1
[ 0.0785 ] Sending psc_bl1
[ 0.1013 ] Sending bct_mb1
[ 0.1140 ] Generating blob for T23x
[ 0.1163 ] tegrahost_v2 --chip 0x23 0 --generateblob blob.xml blob.bin
[ 0.1178 ] The number of images in blob is 20
[ 0.1185 ] blobsize is 84522284
[ 0.1186 ] Added binary blob_uefi_jetson_with_dtb_sigheader.bin.encrypt of size 3584064
[ 0.1669 ] Added binary blob_pscfw_t234_prod_sigheader.bin.encrypt of size 310768
[ 0.1672 ] Added binary blob_mce_flash_o10_cr_prod_sigheader.bin.encrypt of size 187120
[ 0.1676 ] Added binary blob_tsec_t234_sigheader.bin.encrypt of size 176128
[ 0.1679 ] Added binary blob_applet_t234_sigheader.bin.encrypt of size 279616
[ 0.1680 ] Not supported type: mb2_applet
[ 0.1681 ] Added binary blob_mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt of size 439968
[ 0.1683 ] Added binary blob_xusb_t234_prod_sigheader.bin.encrypt of size 164864
[ 0.1685 ] Added binary blob_nvpva_020_sigheader.fw.encrypt of size 2164640
[ 0.1686 ] Added binary blob_display-t234-dce_sigheader.bin.encrypt of size 12065600
[ 0.1758 ] Added binary blob_nvdec_t234_prod_sigheader.fw.encrypt of size 294912
[ 0.1790 ] Added binary blob_bpmp_t234-TE992M-A1_prod_sigheader.bin.encrypt of size 1046080
[ 0.1792 ] Added binary blob_fsi-fw-ecc_sigheader.bin.encrypt of size 5746688
[ 0.1851 ] Added binary blob_tegra234-bpmp-3701-0008-3737-0000_with_odm_sigheader.dtb.encrypt of size 156096
[ 0.1892 ] Added binary blob_camera-rtcpu-t234-rce_sigheader.img.encrypt of size 458096
[ 0.1894 ] Added binary blob_adsp-fw_sigheader.bin.encrypt of size 414976
[ 0.1895 ] Added binary blob_spe_t234_sigheader.bin.encrypt of size 270336
[ 0.1897 ] Added binary blob_tos-optee_t234_sigheader.img.encrypt of size 1600576
[ 0.1899 ] Added binary blob_eks_t234_sigheader.img.encrypt of size 9232
[ 0.1942 ] Added binary blob_boot.img of size 54906880
[ 0.2313 ] Added binary blob_tegra234-p3737-0000+p3701-0008-nv.dtb of size 244540
[ 0.3003 ] tegrarcm_v2 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
[ 0.3008 ] BL: version 1.4.0.2-t234-54845784-08a4de08 last_boot_error: 0
[ 0.3762 ] Sending bct_mem
[ 0.4006 ] Sending blob
[ 8.0134 ] RCM-boot started
Hi,KevinFFF
wpceswpces:
Hi,KevinFFF
When both slots of the device are damaged, the device will enter recovery mode (recovery boot). Doesn’t the device automatically execute the OTA script, like nv_ota_run_tasks.sh? I noticed in the development documentation that OTA is mainly used for upgrading between different SDK versions. I’m wondering if it’s possible to create an offline image that packages my current rootfs and bootloader, using the same SDK version, and then automatically restore both slots once the device enters recovery mode? Could you explain this in more detail? If this is possible, how can I verify it? I noticed that the OTA development documentation doesn’t explain this situation in detail. For the scenario I want to implement, can this OTA process be done remotely on the device without relying on a host computer?"
If OTA can fulfill the requirements I mentioned above, how exactly do I create the offline OTA image package? Can it be placed on an NVMe drive or other storage media?After the device enters recovery mode, How is it associated with nv_ota_run_tasks.sh to automatically trigger the update? After the update, which slot will the device switch to?
How about this question ? May I ask if OTA can solve the redundancy recovery issue independently, without relying on the Host computer?
You can simply remove /lib so that your board should not boot correctly.
To perform an OTA update , you must first boot the device so that you can place the OTA payload (the ota_payload_package.tar.gz file) onto it.
Regarding your assumption, using OTA to recover an unbootable slot is certainly available . For example, if Slot A is unbootable , you can successfully perform the OTA update from Slot B .
Hi,KevinFFF
KevinFFF:
You can simply remove /lib so that your board should not boot correctly.
No, no — I’m in the case with ROOTFS_AB=0, there is only a single rootfs. What I mean is: with respect to the three device nodes /dev/mtd0, /dev/mtd0ro, and /dev/mtdblock0, how can one selectively corrupt the Bootloader of slot A or slot B?
To perform an OTA update , you must first boot the device so that you can place the OTA payload (the ota_payload_package.tar.gz file) onto it.
Regarding your assumption, using OTA to recover an unbootable slot is certainly available . For example, if Slot A is unbootable , you can successfully perform the OTA update from Slot B .
Is this kind of OTA recovery automatic — for example, does Slot B automatically recover Slot A?
If both Slot A and Slot B are damaged, the device will enter recovery mode . In that case, will it also recover automatically?
Which slot will it recover?
And after recovery, which slot will the device boot from?
Please help clarify these questions, thank you.
It will not be recovered automatically.
You have to perform OTA update or Capsule update manually to recover it.
You have to recover it manually.
If it is caused by something removed in rootfs, then your can put that back to recover.
KevinFFF:
What I mean is: with respect to the three device nodes /dev/mtd0, /dev/mtd0ro, and /dev/mtdblock0, how can one selectively corrupt the Bootloader of slot A or slot B?
Please try to mount them and check what are included first.
Hi,KevinFFF
KevinFFF:
It will not be recovered automatically.
You have to perform OTA update or Capsule update manually to recover it.
You have to recover it manually.
If it is caused by something removed in rootfs, then your can put that back to recover.
But I found that after entering recovery mode, the OTA process seems to run automatically.
▒▒verify_rate_range: FMON_PWM3: rate 11027028 below min 18750000
fmon_update_config: FMON_PWM3: detected fault 0x80
▒▒Checking whether device /dev/mmcblk?p1 exist
[ 8.968264] Finding OTA work dir on external storage devices
Looking for OTA work directory on the device(s): /dev/mmcblk0p1
[ 8.972360] mount /dev/mmcblk0p1 /mnt
[ 8.993014] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 8.994645] is_boot_only_partition /mnt
[ 9.006854] OTA work directory /mnt/ota_work is not found on /dev/mmcblk0p1
Checking whether device /dev/sd?1 exist
Device /dev/sd?1 does not exist
Checking whether device /dev/nvme?n1p1 exist
Looking for OTA work directory on the device(s): /dev/nvme0n1p1
[ 9.018290] mount /dev/nvme0n1p1 /mnt
[ 9.031309] EXT4-fs (nvme0n1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 9.032871] is_boot_only_partition /mnt
[ 9.038009] OTA work directory /mnt/ota_work is not found on /dev/nvme0n1p1
[ 9.042578] Finding OTA work dir on internal storage device
[ 9.043774] mount /dev/mmcblk0 /mnt
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mmcblk0, missing codepage or helper program, or other error.
[ 9.082657] Failed to mount /dev/mmcblk0 on the /mnt
[ 9.083821] Failed to run "mount_ota_work_partition /dev/mmcblk0 /mnt"
[ 9.085025] OTA work directory is not found on internal and external storage devices
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.1#
KevinFFF:
What I mean is: with respect to the three device nodes /dev/mtd0, /dev/mtd0ro, and /dev/mtdblock0, how can one selectively corrupt the Bootloader of slot A or slot B?
Please try to mount them and check what are included first.
I’m very sorry,the filesystem cannot be mounted properly.
root@tegra-ubuntu:/mnt# mount /dev/mtd
mtd0 mtd0ro mtdblock0
root@tegra-ubuntu:/mnt# mount /dev/mtd0 test/
mount: /mnt/test: /dev/mtd0 is not a block device.
root@tegra-ubuntu:/mnt# mount /dev/mtd0ro test/
mount: /mnt/test: /dev/mtd0ro is not a block device.
root@tegra-ubuntu:/mnt# mount /dev/mtdblock0 test/
mount: /mnt/test: wrong fs type, bad option, bad superblock on /dev/mtdblock0, missing codepage or helper program, or other error.
root@tegra-ubuntu:/mnt# umount test/
umount: test/: not mounted.
root@tegra-ubuntu:/mnt# cat /proc/mtd
dev: size erasesize name
mtd0: 04000000 00001000 "spi2.0"
root@tegra-ubuntu:/mnt# dmesg | grep mtd
root@tegra-ubuntu:/mnt#
root@tegra-ubuntu:/mnt#
The OTA process will run only if there’s the OTA payload exists and you have to put that payload manually.
May I know what’s your requirement now? To break some partition in QSPI to check if the redundant bootloader mechanism works
Hi,KevinFFF
KevinFFF:
The OTA process will run only if there’s the OTA payload exists and you have to put that payload manually.
If we manually put it in, does it store the payload on the hard drive, and will the OTA automatically locate it or pre-specify it to that location? It shouldn’t depend on the host computer, right?
May I know what’s your requirement now? To break some partition in QSPI to check if the redundant bootloader mechanism works
Yes. I need a complete test procedure: corrupt Slot A, try booting three times and verify it falls back to Slot B; a script should be able to perform the upgrade and recovery, and upon successful recovery switch back to Slot A.
KevinFFF:
If we manually put it in, does it store the payload on the hard drive, and will the OTA automatically locate it or pre-specify it to that location? It shouldn’t depend on the host computer, right?
Please refer to <Linux_for_Tegra>/tools/ota_tools/version_upgrade/Image_based_OTA_Examples.txt for the detailed workflow to perform OTA update. There are many use cases and steps that you can refer to.
Before OTA update, you may need to run the following command to trigger it.
$ sudo ./nv_ota_start.sh ~/ota_payload_package.tar.gz
I would suggest you also enabling rootfs a/b and you can simply remove /lib in rootfs-A so that it will boot failed and boot into slot B after 3 times trials. It is the similar flow as I’ve verified in previous r35 release on T194 series.
Rootfs A/B redundancy fail-over mechanism in Jetpack5.1