We are trying to use rootfs redundancy on the AGX INdustrial version.
Jetpack R32.6.1
We are following the instructions here:
https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/rootfs_custom.html#wwpID0E06E0HA
We see, for example, that these files are present
./bootloader/t186ref/cfg/flash_t194_sdmmc_rootfs_ab.xml
./bootloader/t186ref/cfg/flash_t194_sdmmc.xml
We run this command (per the instructions above)
sudo ./flash.sh ROOTFS_AB=1 jetson-agx-xavier-industrial mmcblk0p1
It fails however to flash both APP and APP_b partitions.
./bootloader/flash.xml is recreated as part of the build process
however it does NOT contain an APP_b partition.
I have included part of the flash log here. You can see it flashes APP. However
it does not flash APP_b.
I can include the complete log if needed.
flash log starts here ----------------------------------------------------------------------------------------------
[ 333.6373 ] Writing partition sce-fw with sce_t194_sigheader.bin.encrypt
[ 335.3627 ] […] 100%
[ 335.3769 ] Writing partition sce-fw_b with sce_t194_sigheader.bin.encrypt
[ 337.7592 ] […] 100%
[ 337.7722 ] Writing partition bpmp-fw with bpmp_t194_sigheader.bin.encrypt
[ 340.1542 ] […] 100%
[ 340.1862 ] Writing partition bpmp-fw_b with bpmp_t194_sigheader.bin.encrypt
[ 345.5801 ] […] 100%
[ 345.6113 ] Writing partition bpmp-fw-dtb with tegra194-a02-bpmp-p2888-0008-b01-jaxi_sigheader.dtb.encrypt
[ 351.0065 ] […] 100%
[ 351.0124 ] Writing partition bpmp-fw-dtb_b with tegra194-a02-bpmp-p2888-0008-b01-jaxi_sigheader.dtb.encrypt
[ 352.1721 ] […] 100%
[ 352.1785 ] Writing partition CPUBL-CFG with cbo.dtb
[ 353.3374 ] […] 100%
[ 353.3389 ] Writing partition CPUBL-CFG_b with cbo.dtb
[ 353.3470 ] […] 100%
[ 353.3488 ] Writing partition VER with qspi_bootblob_ver.txt
[ 353.3571 ] […] 100%
[ 353.3768 ] Writing partition VER_b with qspi_bootblob_ver.txt
[ 353.3852 ] […] 100%
[ 353.3871 ] Writing partition master_boot_record with mbr_1_3.bin
[ 353.3947 ] […] 100%
[ 353.3963 ] Writing partition APP with system.img
[ 353.4045 ] […] 100%
[ 671.3584 ] Writing partition kernel with boot_sigheader.img.encrypt
[ 671.4186 ] […] 100%
[ 673.5871 ] Writing partition kernel_b with boot_sigheader.img.encrypt
[ 673.6153 ] […] 100%
[ 675.7940 ] Writing partition kernel-dtb with kernel_tegra194-p2888-0008-p2822-0000_sigheader.dtb.encrypt
[ 675.8115 ] […] 100%
[ 675.8214 ] Writing partition kernel-dtb_b with kernel_tegra194-p2888-0008-p2822-0000_sigheader.dtb.encrypt
[ 675.8440 ] […] 100%
[ 675.8543 ] Writing partition recovery with recovery_sigheader.img.encrypt
[ 675.8760 ] […] 100%
[ 678.4863 ] Writing partition recovery-dtb with tegra194-p2888-0008-p2822-0000.dtb_sigheader.rec.encrypt
[ 678.4951 ] […] 100%
Hi,
Are you saying that the flash directly goes failure? Could you also test new jetpack release like jp4.6.1?
HI WayneWWW,
It flashes successfully but does not create a 2nd rootfs partition (APP_b).
I have attached the full log showing this.
I can test with a newer version of Jetpack but our problem is that we have a driver
dependency on R32.6.1
flash-dual-rootfs-but-cmd-fails.txt (66.4 KB)
Can you move the before flash.sh but not after flash.sh?
$ sudo ROOTFS_AB=1 ./flash.sh xxxxx mmcblk0p1
Ok, that helps to get it flashed. As you see from
LOG1–flash-dual-rootfs-success.txt
both APP and APP_b are flashed.
However it fails to boot. I see a kernel panic.
ref --
LOG2--boot--kernel-panic.txt or
LOG2--boot--kernel-panic--TEST2.txt
I then reflashed without rootfs redundancy.
> sudo ./flash.sh jetson-agx-xavier-industrial mmcblk0p1
but I still see the same kernel panic on boot.
I then reflashed with the SDK Manager (GUI)
Now I am able to boot again from eMMC
I have tried this same procedure 2 times with the same result.
Let me know what you think.
LOG1–flash-dual-rootfs-success.txt (77.7 KB)
LOG2–boot–kernel-panic.txt (7.9 KB)
LOG2–boot–kernel-panic–TEST2.txt (32.9 KB)
Is that the jetson AGX devkit?
yes, AGX INdustrial version
Looks weird. Sorry that could you also test the 32.7.1 version to prevent any bug on 32.6.1?
Just for a test.
I get the same result using R32.7.1
Steps.
-
updgrade to Jetpack 4.6.1
-
sudo ROOTFS_AB=1 ./flash.sh jetson-agx-xavier-industrial mmcblk0p1
see log
successfully creates APP and APP_b partitions.
-
boot fails
kernel panic.
see log (it seems to only find 1 partition to boot from).
LOG3–flash-dual-rootfs-success-R32_7_1.txt (77.8 KB)
LOG3–boot–kernel-panic_R32_7_1.txt (29.8 KB)
Hi,
Can we firstly try the basic case? Since you said you would hit issue even w/o using rootfsAB, could you flash it with the base case first? I mean just common flash, no rootfsAB and something else. Just same as how sdkmanager flash it.
yes, you are right. I have done the following.
- got it in a known bootable state using SDK Manager
- then flashed it again
sudo ./flash.sh jetson-agx-xavier-industrial mmcblk0p1
It fails with the same kernel panic.
I think the installation must be corrupted somehow.
I will do a complete reinstall and try again.
Hi WayneWWW,
After a reinstall here are some observations.
Using Jetpack 4.6; R32.6.1
#1.
LOG4-flash-after-reinstall (see attached)
1. shows APP and APP_b are flashed
LOG4–flash-after-reinstall.txt (83.4 KB)
2. Shows SMD and SMD_b are flashed
#2
However only 1 partition is available on boot.
I> Validate extlinux.conf …
I> T19x: Authenticate extlinux.conf (bin_type: 54), max size 0x2000
I> Encryption fuse is not ON
I> L4T boot options
I> [1]: “primary kernel”
I> Enter choice:
[0012.975] I> Invalid option: 2
[0012.975] I> Enter choice:
[0013.843] I> Invalid option: 3
[0013.843] I> Enter choice:
[0015.817] I> Selected option: 1
#3
The image will boot correctly. (Before the full Jetpack reinstall this is where it failed
with a kernel panic).
It looks like it created 2 rootfs partitions as the full EMMC size is 32GB but
only about 50% is used for the mounted rootfs.
df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mmcblk0p1 14384136 5290868 8342884 39% /
none 13894252 0 13894252 0% /dev
tmpfs 14278940 4 14278936 1% /dev/shm
tmpfs 14278940 21652 14257288 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 14278940 0 14278940 0% /sys/fs/cgroup
tmpfs 2855788 12 2855776 1% /run/user/120
tmpfs 2855788 0 2855788 0% /run/user/1000
#4
The SMD partitions look like they are corrupt.
merlin@merlin-desktop:~$ nvbootctrl dump-slots-info
Fail to open metadata file
Init SMD partition failed!
Fail to open metadata file
Fail to open metadata file
merlin@merlin-desktop:~$ ��
merlin@merlin-desktop:~$ nvbootctrl -t rootfs is-unified-enabled
Fail to open metadata file
Init SMD partition failed!
Fail to open metadata file
Fail to open metadata file
Error querying the status of unified a/b: Input/output error
Again, could you test this with 32.7.1 and see if this issue is still there?
I have attached logs but basically I see the same problem with R32.7.1
It appears to flash successfully. And the partition again is approx 50% of the physical capability
of the 32GB emmc part.
merlin@merlin-desktop:~$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mmcblk0p1 14384136 5325752 8308000 40% /
none 13894252 0 13894252 0% /dev
tmpfs 14278940 4 14278936 1% /dev/shm
tmpfs 14278940 31372 14247568 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 14278940 0 14278940 0% /sys/fs/cgroup
tmpfs 2855788 0 2855788 0% /run/user/0
tmpfs 2855788 20 2855768 1% /run/user/120
tmpfs 2855788 0 2855788 0% /run/user/1000
But I can only boot from 1 partition
T19x: Authenticate extlinux.conf (bin_type: 54), max size 0x2000
I> Encryption fuse is not ON
I> L4T boot options
I> [1]: “primary kernel”
I> Enter choice:
[0018.858] I> Continuing with default option: 1
[0018.858] I> Loading kernel …
[0018.858] I> Loading kernel binary from rootfs …
[0018.859] I> rootfs path: /sdmmc_user/boot/Image
And the nvbootctrl commands don’t work.
merlin@merlin-desktop:~$ nvbootctrl -t rootfs get-current-slot
Fail to open metadata file
Init SMD partition failed!
Fail to open metadata file
Fail to open metadata file
0
merlin@merlin-desktop:~$
LOG5–flash-r32-7-1.txt (83.4 KB)
LOG5–boot–r32-7-1.txt (13.7 KB)
Hi,
We just tried to use our AGXi devkit and try to enable rootfsAB, but we cannot reproduce any error. Also, the slot could be switched too.
Can you firstly check /dev/disk/by-partlabel and see if there is APP_b on your device?
yes, it is there.
The error complains about the SMD partition. Although the log shows (previous upload) it flashed it.
Hi,
Can we clarify the version is correct first? In your previous comment, you told me you are flashing 32.7.1, but the host side log you shared is still using jp4.6, which is 32.6.1…
you’re right I didn’t realize it created a new directory for 32.7.1.
The results are the same however.
LOG6–flash-r32-7-1.txt (83.5 KB)
There is only 1 partition to boot from.
Again the rootfs partition seems to be present in /dev/disks but the SMD partition looks corrupted
LOG6–boot–r32-7-1.txt (33.1 KB)
Here you go – (I think these are complete; they are just copied out of the minicom window).
thanks !
LOG6–dmesg----r32-7-1.txt (94.8 KB)
LOG6–nv_boot_ctrl_conf–r32-7-1.txt (306 Bytes)
LOG6–flash-r32-7-1.txt (83.5 KB)
