A/B Rootfs Redundancy: No fallback to other slot when one gets corrupted

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi all,

I’m currently working with a Jetson Orin running JetPack 5.1.2 and encountering issues with A/B rootfs redundancy not behaving as expected during a failure scenario.

Im experiencing same issue as the user in this thread: A/B ROOTFS Redundancy: Bootloader does not boot from backup slot when the working slot is intentionally corrupted
but instead of running jetpack 4.6 im running 5.1.2.

I can successfully boot from either slot A or slot B by using nvbootctrl set-active-boot-slot 0 or 1

But when I corrupt slot B by writing 10MB of zeroes to the rootfs partition:

sudo dd if=/dev/zero of=/dev/nvme0n1p2 bs=1M count=10

Set slot B active:

sudo nvbootctrl set-active-boot-slot 1

Reboot.

It keeps booting into slot B and gets stuck on exFAT-fs errors. Even If I unpower it and boot it again 3+ times it doesn’t switch back to slot A.
boot.txt (123.8 KB)

Am I understanding it completely wrong that it should automatically switch to other slot after 3 failed attempts, am I missing something or has something changed in JP5.1.2?

Im flashing it with ROOTFS_AB=1 and using config flash_l4t_nvme_rootfs_ab.xml

Also I noticed these commands give me not implemented messages is that okay?

$ sudo nvbootctrl is-rootfs-ab-enabled
is-rootfs-ab-enabled is not implemented.
$ sudo nvbootctrl verify
Info: variable BootChainFwStatus is not found.
3

hello daniel.banar,

this is an incorrect approach for testing rootfs redundancy.
the retry_count decremented is triggered by software (i.e. warm-reboot).
unpower or using a physical button is cold reset. you should have warm reset, such as $ sudo reboot commands to reboot the system.

you may see-also similar discussion threads, such as Topic 301119, or Topic 332725 for reference.

is it a must to stay-on JP-5.1.2/r35.4.1?
is it possible for moving forward to the latest release version (i.e. JetPack 6.2) for verification?

4

Hi @daniel.banar , have a look at the following thread:

I think corrupting a big chunk (such as 10MB) breaks the rootfs beyond repair (for some reason, it doesn’t trigger the failover mechanism). Try removing only a portion of rootfs (rm -rf) as mentioned in the above thread. I have not tried this with Orin, but it works well enough on AGX Xavier (on some Jetpack releases).

5

Hi I mounted the partition rm -rf ed it and at first it didnt do anything, just got stuck on kernel panic, after 2 minutes it rebooted and then after 2 more it switched back to slot A and B was marked as unbootable.

Still, is this really the best it can do? I was looking for something that, after power-up, fully boots and then marks the boot as successful. If that doesn’t happen—say, it crashes or gets stuck—it should detect that and automatically switch to the fallback. Is that kind of behavior possible to configure?

6

hello daniel.banar,

did you meant you would like to reduce the retry counts? it’s by default to have 3-trails on slot-A and then fail-over to slot-B.
please try re-flash your target by adding the ROOTFS_RETRY_COUNT_MAX to configure the maximum retry times for rootfs.
note, the valid value of retry counts is 0 to 3.
for instance,
$ sudo ROOTFS_AB=1 ROOTFS_RETRY_COUNT_MAX=3 ./flash.sh [options] <target_board> <rootdev>

7

Not really Im looking for a way to use cold boots and not rely on the watchdog to reboot it or decide when the watchdog actually marks it as successful boot

8

hello daniel.banar,

unfortunately, it’s not supported. the retry_count decremented is triggered by software (i.e. warm-reboot).

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.