L4T Version: JetPack 5.x (R35.x branches, using NVIDIA UEFI)
Boot Media: Internal eMMC (mmcblk0)
Security: Full-Disk Encryption (FDE) enabled via eKB and OP-TEE
Description of the Issue:
Following a sudden power off/ power cycle, an AGX Xavier unit entered a terminal boot loop, eventually dropping into the UEFI Interactive Shell v2.2.
Initially, we suspected a hardware communication or PMIC failure due to severe I2C errors during the MB2 stage. However, by comparing the broken boot log side-by-side with a known healthy log from an identical working board, we made a crucial discovery: The early stage I2C errors occur identically on the good board and are completely benign. The true root cause is severe file allocation table/metadata corruption on the unencrypted FAT32 EFI System Partition (ESP / fs2:), which has truncated critical boot assets down to double-digit bytes.
Diagnostic Logs & Key Findings:
1. The Red Herring Ruled Out (Early I2C Failures): Both our working and broken boards show the exact same failure when attempting to read the CVB EEPROM at address 0xAE during the MB2 (TBoot-BPMP) phase. This hardware behavior can be safely ignored:
Plaintext
[0001.213] E> I2C: slave not found in slaves.
[0001.214] E> I2C: Could not write 0 bytes to slave: 0x00ae with repeat start true.
[0001.217] E> eeprom: Failed to read I2C slave device
[0002.852] E> Failed to get board id info!
2. FAT32 Corruption on fs2::
The actual divergence occurs when L4TLauncher fails with Android image header not seen. Dropping into the UEFI Interactive Shell and executing ls on the unencrypted ESP partition (fs2:) reveals that the filesystem metadata has collapsed. The allocation sizes for the core boot images have been completely truncated:
Image → 12 bytes (Should be ~20+ MB)
initrd → 13 bytes (Should be ~10+ MB)
extlinux.conf → 29 bytes
3. UEFI Shell Core Crash: When attempting to view the text inside the truncated extlinux.conf file, the malformed allocation table structure triggers a hard firmware assertion panic inside the EDK II command library, forcing a full system reset:
Shell> fs2:
FS2:\> type extlinux.conf
type: Operation was not successful on 'extlinux.conf'
ASSERT [Shell] /home/rr/build/nvidia-uefi-r35.6.1/edk2/ShellPkg/Library/UefiShellLevel3CommandsLib/Type.c(304): ShellStatus == SHELL_SUCCESS
Resetting the system in 5 seconds.
Because the A/B redundancy slots share this same corrupted ESP partition layout, both Slot 0 and Slot 1 fail similarly, leaving the board unbootable.
Questions:
ESP Reconstruction without Re-flashing: We have an active, encrypted RootFS partition that we must preserve for forensic debugging. Since the hardware is healthy and the corruption is isolated entirely to the unencrypted FAT32 FS2 partition, is there a supported method to rebuild, format, or fix the allocation tables of the ESP partition directly from the UEFI shell or via a non-destructive custom flash.sh command target?
Type.c (304) Assertion Origin: What structural state inside a corrupted FAT32 directory entry causes the EDK II shell library to trigger this exact hard assertion panic on line 304 of Type.c?
Production Mitigations against FAT32 Corruption: Because the default Jetson partition layout relies heavily on a standard FAT32 structure for the unencrypted boot layers, sudden power losses present a high risk of metadata truncation. For remote field deployments utilizing Full-Disk Encryption, what is the recommended architecture to protect this space? Can the ESP layer be safely mounted as read-only by the Linux kernel during production runtime to prevent mid-write corruption?
The early MB2 I2C errors seem not like the root cause. The issue is more likely ESP/FAT32 corruption on eMMC.
If the board can still enter recovery kernel shell, then yes, ESP can be repaired there without reflashing the encrypted rootfs.
We sugggest to back up the current ESP first, then restore a known-good esp.img or known-good ESP contents with the same BSP configuration, restore the required boot variables, and reboot.
Please refer to the following steps to update BOOTAA64.efi first.
(if esp partition is on `/dev/mmcblk0p10`)
sudo mount /dev/mmcblk0p10 /mnt
sudo cp BOOTAA64.efi /mnt/EFI/BOOT/BOOTAA64.efi
sync
sudo umount /mnt
Hi KevinFFF, thank you for the quick confirmation and ruling out early MB2 hardware/I2C issues and isolating this strictly to ESP/FAT32 corruption on the eMMC.
You mentioned repairing the ESP via the recovery kernel shell without re-flashing. Unfortunately, the corruption on this unit appears to have impacted the entire partition uniformly. The system automatically attempts a recovery boot sequence but fails immediately with an empty header panic:
L4TLauncher: Attempting Recovery Boot
Android image header not seen
Failed to boot recovery:0 partition
Because the recovery kernel stubs are also unreadable, we cannot drop into a local recovery terminal to manually run the mount and cp commands on /dev/mmcblk0p1 (our ESP partition mapped as FS2).
I have attached our serial boot log showing the loop behavior, the mapping table, and the exact EDK2 Type.c(304) firmware assertion crash for your reference.
To help us secure our environment and prevent this from happening to other devices deployed in the field, we would appreciate insight into the following:
What caused this specific truncation? Looking at the directory listing on fs2:, the file metadata pointers are entirely broken—the file entries exist, but Image is truncated to 12 bytes, initrd to 13 bytes, and extlinux.conf to 29 bytes. Is this clean truncation a known behavior of how the L4T filesystem syncs or flushes data during runtime operations? Was the system actively writing to the ESP space at the time of the crash, or is this standard FAT32 vulnerability to possible sudden voltage drops?
What are the best practices to prevent ESP corruption in production field units? Since our primary RootFS is fully encrypted, protecting the unencrypted bootloader components is a high priority.
Can the ESP partition (/dev/mmcblk0p1) be safely mounted as Read-Only (ro) in Linux via /etc/fstab during production runtime to completely immunize it against sudden power-cut corruption?
Does NVIDIA update or alter the ESP contents during normal runtime operations, or is it strictly read during the UEFI initialization phase?
Alternative Non-Destructive Repairs: Given that we cannot boot into the local recovery kernel but can access the UEFI Interactive Shell and hardware Force Recovery Mode (RCM), is there a way to flash/overwrite only the A_esp partition via a specific flash.sh target sequence on the host PC without wiping or modifying the encrypted user data blocks?
firstly, why there’s a sudden power-off/power cycle to introduce the issue?
according to Product Design Guide, there should be protection of the sudden power lose.
may I know what’s the reproduce steps for reference? can you reproduce the same on the Jetson AGX Xavier developer kit?
secondly, it did support flashing a single partition with the -k options.
see-also developer guide, Flashing a Specific Partition.
for instance, $ sudo ./flash.sh -r -k esp --image bootloader/esp.img jetson-agx-xavier-devkit mmcblk0p1
Thanks for following up and for the pointer on the -k flag. We appreciate the support. Here is an update on our investigation into the fatal boot errors, along with a few areas where we would love to get your recommendations to confirm our next steps.
1. Sudden Power-Off and Hardware Protection
Regarding the sudden power-off: our devices are deployed in environments where unexpected power severing can occur. While we understand the Product Design Guide (PDG) outlines protections, we are essentially dealing with hard-cut scenarios.
We understand the AGX architecture is complex and pulling the power abruptly is prone to causing filesystem corruption, particularly on the FAT32 ESP partition.
Hardware Solution: We agree with your previous suggestion that a hardware addition (such as a backup supercapacitor or a dedicated graceful shutdown circuit) is likely required to intercept the power loss and safely halt the system. Before we finalize our design, we want to ensure we are aligned with NVIDIA’s best practices. Could you provide any specific reference designs, recommended power-management ICs, or official guidelines on how NVIDIA recommends implementing this hardware-level power protection? Having your confirmation on a proven approach would be incredibly helpful.
Reproduction: For your reference, the issue can be reproduced by repeatedly hard-cycling the power supply during active read/write operations or boot phases, simulating a total grid failure.
2. UEFI Bug in Jetpack 5.1.5
During our investigation, we also isolated a critical UEFI bug present in Jetpack 5.1.5.
The Issue: We found a logic error in the component managing UEFI variables, which causes data corruption within the QSPI NOR flash during garbage collection or variable updates. This corruption causes the bootloader to hit a fatal error and freeze the system entirely before it can reach the OS.
The Fix: We see that NVIDIA addressed this in Jetpack 5.1.6. Can you confirm if this aligns with your internal findings, and if there are any additional patches or configurations in 5.1.6 we should apply to fully mitigate this QSPI corruption?
3. Partition Flashing (-k option)
Thank you for highlighting the $ sudo ./flash.sh -r -k esp command. It is helpful for restoring the truncated bootloader partitions without having to reflash the entire mmcblk0p1 drive, though our ultimate goal remains preventing the corruption at the hardware level.
We look forward to your recommendations on the hardware protection design.