Hi, we are working on Jetson Orin nano nvme and meta-tegra, we have enabled A/B redundant rootfs and we are able to switch between the two partitions with nvbootctrl.
Nevertheless we attempted to verify the automatic recovery of the redundant rootfs by corrupting rootfs A with dd (while on B), switching partition and reboot, Similar to : https://forums.developer.nvidia.com/t/orin-nano-ab-booting-not-working/301119/12?u=daniel.gonzalez3 but the systems never resets.
We noticed watchdog was “disabled” and we set it to “okay” following: https://forums.developer.nvidia.com/t/jetpack-5-1-kernel-panic-does-not-lead-to-reboot-with-a-b-system/245215/1
But this didnt’ work also it gave the next errors in dmesg related to the watchdog:
```
root@jetson-orin-nano-devkit-nvme:~# dmesg | grep -i tegra_wdt
[ 10.468126] tegra_wdt_t18x 2190000.watchdog: can’t request region for resource [mem 0x02190000-0x0219fffe]
[ 10.468135] tegra_wdt_t18x 2190000.watchdog: Cannot request memregion/iomap res_wdt
[ 10.468137] tegra_wdt_t18x: probe of 2190000.watchdog failed with error -16
root@jetson-orin-nano-devkit-nvme:~# journalctl -b | grep -i watchdog
May 29 18:48:23 jetson-orin-nano-devkit-nvme kernel: tegra_wdt_t18x 2190000.watchdog: can’t request region for resource [mem 0x02190000-0x0219fffe]
May 29 18:48:23 jetson-orin-nano-devkit-nvme kernel: tegra_wdt_t18x 2190000.watchdog: Cannot request memregion/iomap res_wdt
May 29 18:48:23 jetson-orin-nano-devkit-nvme kernel: tegra_wdt_t18x: probe of 2190000.watchdog failed with error -16
May 29 18:49:28 jetson-orin-nano-devkit-nvme kernel[690]: [ 10.468126] tegra_wdt_t18x 2190000.watchdog: can’t request region for resource [mem 0x02190000-0x0219fffe]
May 29 18:49:28 jetson-orin-nano-devkit-nvme kernel[690]: [ 10.468135] tegra_wdt_t18x 2190000.watchdog: Cannot request memregion/iomap res_wdt
May 29 18:49:28 jetson-orin-nano-devkit-nvme kernel[690]: [ 10.468137] tegra_wdt_t18x: probe of 2190000.watchdog failed with error -16
```
Which its very similar to the one presented here: https://forums.developer.nvidia.com/t/rootfs-ab-enable-but-cannot-reboot-when-a-boot-fail/335958/21
There the issue was fixed by applying some patches to the kernel, although it is for R36.3 and we are on R36.4.4 and the patches could be applied, we tried this and at the moment is still not working,
We’d like to ask if there are appropriate patches for R36.4.4 or if anyone could provide any other insight about this issue ?
a_corrupted_test.txt (113.9 KB)
hello daniel.gonzalez3,
may I also know what’s your steps to corrupt a rootfs slot?
and.. what’s your approaches to attempt system reboot to trigger the test.
Hi @JerryChang thanks for the answer, we used dd to corrupt A while on B:
root@jetson-orin-nano-devkit-nvme:~# dd if=/dev/zero of=/dev/nvme0n1p1 bs=1M count=1
1+0 records in
1+0 records out
And then switch to A and reboot:
root@jetson-orin-nano-devkit-nvme:~# nvbootctrl set-active-boot-slot 0
root@jetson-orin-nano-devkit-nvme:~# nvbootctrl dump-slots-info
Current version: 36.4.4
Capsule update status: 0
Current bootloader slot: B
Active bootloader slot: A
num_slots: 2
slot: 0, status: normal
slot: 1, status: normal
root@jetson-orin-nano-devkit-nvme:~# reboot
And boot attemped gets stuck here:
[ 7.360887] pci_bus 0007:00: root bus resource [io 0x300000-0x3fffff] (bus address [0x3e100000-0x3e1fffff])
[ 7.360890] pci_bus 0007:00: root bus resource [mem 0x3228000000-0x322fffffff] (bus address [0x40000000-0x47ffffff])
[ 7.360893] pci_bus 0007:00: root bus resource [bus 00-ff]
[ 7.360895] pci_bus 0007:00: root bus resource [mem 0x3000000000-0x3227ffffff pref]
[ 7.360937] pci 0007:00:00.0: [10de:229a] type 01 class 0x060400
[ 7.361110] pci 0007:00:00.0: PME# supported from D0 D3hot
[ 7.366460] pci 0007:00:00.0: PCI bridge to [bus 01-ff]
[ 7.366542] pcieport 0007:00:00.0: Adding to iommu group 6
[ 7.366615] pcieport 0007:00:00.0: PME: Signaling with IRQ 194
[ 7.367153] pcieport 0007:00:00.0: AER: enabled with IRQ 194
[ 7.367356] pci_bus 0007:01: busn_res: [bus 01-ff] is released
[ 7.367429] pci 0007:00:00.0: Removing from iommu group 6
[ 7.367438] pci_bus 0007:00: busn_res: [bus 00-ff] is released
[ 7.465783] tegra-xudc 3550000.usb: Adding to iommu group 7
[ 7.646641] nvme 0004:01:00.0: Adding to iommu group 5
[ 7.646871] nvme nvme0: pci function 0004:01:00.0
[ 7.646927] nvme 0004:01:00.0: enabling device (0000 → 0002)
[ 7.650475] nvme nvme0: missing or invalid SUBNQN field.
[ 7.661789] nvme nvme0: allocated 64 MiB host memory buffer.
[ 7.672577] nvme nvme0: 6/0/0 default/read/poll queues
[ 7.680336] nvme0n1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13 p14 p15 p16
You can see full log ni the attached filie.
a_corrupted_test.txt (113.9 KB)
hello daniel.gonzalez3,
please see-also Topic 335958 for the steps we’ve tested root file system redundancy,
you should reboot the system after corrupt a slot, instead of switching slots manually.
Hi @JerryChang ,
We applied the patches provided on Topic 335958, then flashed the board, we still get the watchdog error we reported at the beginning of this topic, even though the watchdog status is “okay“. We still decided to test as you requested by corrupting with dd on partition A, then rebooting, and we get the same log as in the response above.
Have you tested the recovery with L4T 36.4.4?
hello luis.merayo,
it’s working on Orin-Nano + r36.4.4
we’ve tested Rootfs-A/B redundancy locally with Orin Nano/ JetPack 6.2.1/r36.4.4 for confirmation.
system shall reboot several times (3-time retry counts by default) after slot-A partition has erased, and then trigger Rootfs-A/B redundancy for entering slot-B.
you should also use $ sudo nvbootctrl -t rootfs dump-slots-info for checking slot status, the slot-A status should be unbootable.