We have several Xavier AGX units stuck in recovery mode. We would like to understand what criteria the bootloader uses to determine when it should put the SoC into recovery mode.
I guess you are talking about recovery boot shown in the UEFI log but not really recovery mode.
Or you have no idea what I am trying to tell here?
Are you using “lsusb” to decide it is in recovery mode? If so, then that is not a valid test. If lsusb does not show the device, then the device cannot be in recovery mode; the converse is not true since a normal boot also will show up with lsusb.
In recovery mode the Jetson becomes a custom USB device understood only by the custom USB driver (in flash software this is conveniently named the “driver package”). In a default normal boot there is software running under the “gadget” kernel framework to pretend to be a virtual router and a virtual mass storage device. These programs are not differentiated from recovery mode, but are ordinary software and not actual recovery mode.
Knowing how you determined that the units are stuck in recovery mode is a non-trivial question. Seeing a serial console log during power up often will answer the question though.
We access it via the UART and land into a bash shell. This is the log message we receive when we run the reboot command from the shell.
L4TLauncher: Attempting Recovery Boot
That is recovery boot image but not recovery mode.
Please follow this to restore and then check if any crash in your normal boot.
To add clarity, when the filesystem cannot be found during an otherwise normal boot it will drop into a bash shell to give an opportunity to rescue the system. Recovery mode is quite different, this is a special mode for making the Jetson behave as a custom USB device. As mentioned, attempting recovery boot is a normal boot which broke part way in and could not find the filesystem. If this uses the A/B recovery partition (not all installs have this set up), then it will try to find the replacement filesystem partition. If there is no A/B recovery option, then it is waiting for the user to do something to repair or make available the correct partition. This is a failed boot.
Did this system ever boot correctly? What storage media does it use, e.g., is it a commercial module with eMMC? Is it a dev kit with an SD card and no eMMC? Those answers change storage media and rootfs details.
Thank you! I’ll be reviewing those resources.
Yes, these units have booted previously but now, for some reason, continuously land back into the recovery boot image. This issue occurs for units in the field that fail at some point during their life cycle but were booting properly before the failure. We did not implement the A/B partition scheme so we have no fallback mechanism in place for recovery. Regarding storage, of the system image, we are using the builtin eMMC storage on the Tegra for Linux. What I would like to do is to disable the automatic soft reset into the recovery boot image so that I can see where it is failing during the Linux boot. I’m wondering if the system image is corrupted because after re-flashing the unit we can typically recover these units.
Do you have direct physical access, and if so, do you have a host PC with you? I’m guessing you do since you said re-flashing fails. What would be interesting is to put the unit in actual recovery mode (temporarily making it a custom USB device) and cloning the root filesystem (which takes a lot of disk space and time). Then examining this on loopback to see what the filesystem actually has going. One can mount a raw clone on loopback (read-only if desired) and see if the filesystem is corrupt or not, what the partition ID is, so on. Ultimately though, a serial console boot log would be the starting place (and if this is local to you, then this is trivial to obtain).
Note: It is also useful if you just flashed to examine the loopback mounted flash image from “Linux_for_Tegra/bootloader/system.img.raw”, e.g., finding its ID and comparing to what the serial console says the ID is which it wants.
Just to clarify, we are able to re-flash these units without issue and that is currently the only way we can currently recover them. However, we would like to know what it causing these failures in the first place so that we can take proactive measures to correct it.
Also, I was able to create a dump of the system image and mount it using the following command sudo ./flash.sh -r -k APP -G ~/raw.img -k APP jetson-xavier mmcblk0p1 however nothing in particular stood out when I looked at the dmesg log files of the last successful system boot of the unit. However, because re-flashing fixes the issue, I’m thinking that something is getting corrupt in the image?
I will take a look at bootloader system.img.raw. Where can I look in this image to get its id? Is there any reference you could point me to?
Thanks for the help! Greatly appreciate it!
The suggestion for cloning the rootfs is in fact to see what is there…it is a forensic method of looking at the failing partition. You cannot boot to and examine a defective partition, but you can loopback mount and examine a raw clone. Assuming the raw clone mount is set to read-only, then messages about mount issues on the host PC become a comment on what caused the failure…if it was the partition or if it was something outside the partition. If not the partition, then you can examine things like partition UID and compare it to the serial console boot log to see if the partition ID was ever mentioned. It is like a game of darts and you will be looking at things related to booting that specific partition image, but if you don’t have the exact image which failed, then you’re using the wrong dart board. It might end up that the issue is in the initrd, but unless your security fuses are burned, then it is likely the initrd is also present on the cloned image in “/boot”.
Do note that when you clone to “raw.img” it will also create “raw.img.raw”. The name you specified as “raw.img” is a sparse image, and the actual raw image is silently created. In fact the raw image is the first clone (the raw.img.raw using the above naming), and then the sparse image is created with the mksparse and NULL filler bytes by reading the true raw image. I tend to throw away the sparse image and keep only the raw image; then, if I am going to store it, use “bzip2 -9” on it (which can take hours to finish). Other than flashing there isn’t a lot you can do with the sparse image.
I will assume your file name for the raw image is “raw.img.raw”. An example to mount read-only, which automatically covers it with a loopback device, assuming you are in the directory with the raw clone:
sudo mount -o loop,ro raw.img.raw /mnt
You can look at the list of loopback devices and determine which one is used via:
losetup --list
I will pretend this device is “/dev/loop2” (very likely it is not actually that device). You can then examine it as if it is a disk. Examples:
sudo gdisk -l /dev/loop2lsblk -f /dev/loop2sudo blkid /dev/loop2(this is the one you want to compare to the serial console boot logs)
If you don’t know what ID is being searched for during boot via serial console, then it is difficult to know if this is an issue or not. I will also highly recommend you log the first mount in case there are error messages:
sudo mount -o loop,ro raw.img.raw /mnt 2>&1 | tee log_loop_mount.txt
(you could just copy and paste or attach that file to the forum thread…but only bother with that if there is an error; we don’t need the log of mount if there is no error)
You can also pull files from this read-only mount. The “/mnt/boot/extlinux/extlinux.conf” file (assuming it is mounted on “/mnt”) would be important. So too would the initrd there. If for some reason the initrd is failing and it is using the partition and not the one in /boot, then we could clone the initrd partition itself and examine that. However, this is a lot more guesswork than if you have the serial console boot log and know exactly what to look for right at the start.