Hey @KevinFFF . It happens from 35.6.2 to 35.6.2 too. I’ll add the debug uefi and send you the logs.
UEFI Log after failing the update by pulling the power(slot marked as normal, while it should be unbootable):
boot_after_bootloader_fail.log (225.4 KB)
UEFI Log after switching the slots with nvbootctrl afterwards without updating (Switched automatically back, because it detected the unbootable slot then)
boot_after_bootloader_fail_switched.log (222.3 KB)
UEFI Log after updating the bootoader again: (Switched the slot, but the other is marked unbootable now)
boot_after_bootloader_update_while_unbootable.log (224.8 KB)
After reboot:
Current version: 35.6.2
Capsule update status: 1
Current bootloader slot: B
Active bootloader slot: B
num_slots: 2
slot: 0, status: unbootable
slot: 1, status: normal
After repower:
nvbootctrl dump-slots-info
Current version: 35.6.2
Capsule update status: 1
Current bootloader slot: B
Active bootloader slot: B
num_slots: 2
slot: 0, status: normal
slot: 1, status: normal
Capsule update status always shows 1, even though it does not update anymore.
Hi seeky15,
We can reproduce similar behavior on the Xavier NX devkit with r35.6.2.
It seems some bit in scratch register does not been cleared. (currently, the HW reset will do that)
The issue is specific to Xavier series(T194).
I will update to you once we get more findings.
Hi seeky15,
We’ve identified the wrong state issue caused from the fail bit marked as 1 in SR due the previous power loss during update.
Could you try running the following command in this step? (i.e. before you perform capsule update from A to B)
$ sudo nvbootctrl set-SR-BR 0
That seems to fix the issue, thank you!
I guess you noticed that too during your tests. When setting the set-SR-BR 0 additionally to updating the bootloader, the system sporadically (around 30-50%) fails to switch the slots.
The systems update the bootloader, then take long time to reboot and afterwards the system comes up with the old slot. The other slot shows the other data, capsule update reports a 1, but system boots from the old one.
Can we prevent that?
You just need to run that command before performing capsule update.
(i.e. you don’t need to run it every time when you switch slot)
I added it right before writing into the OsIndications, so it is only present when the capsule update is running
What’s the result of sudo nvbootctrl dump-slots-info at this moment?
Is there any error log with the process?
In the meantime I have remved the set-SR-BR 0 since it had no value for us.
The issue that after an update sporadically the old slot is still used, despite the bootloader update being successful just now happened again. So it seems like the update sporadically fails to switch the rootfs slot.
Unfortunately this was on a production system now and I do not have any UEFI debug log or capsule update status to share.
Did you encounter such an issue too?
Is your current issue happening on specific board? (i.e. the reproduce rate is not 100%?)
If so, I would need you helping to clarify the difference between the pass and failed case.
Please let us know the different in their logs.
Sorry that we cannot reproduce the similar issue as yours.
Our verification steps are as following:
Step 1. Flash r35.6.2 on Xavier-NX emmc
Step 2. Generate capsule payload from host PC
Step 3. Capsule update from A to B, remove power cable during update
Step 4. Plug-in power cable and boot up
$ sudo nvbootctrl dump-slots-info
Current version: 35.6.2
Capsule update status: 1
Current bootloader slot: A
Active bootloader slot: A
num_slots: 2
slot: 0, status: normal
slot: 1, status: normal
Step 5. Switch to slot B
$ sudo nvbootctrl dump-slots-info
Current version: 35.6.2
Capsule update status: 1
Current bootloader slot: A
Active bootloader slot: A
num_slots: 2
slot: 0, status: normal
slot: 1, status: unbootable
Step 6. Run set-SR-BR before capsule update to clean up SR
$ sudo nvbootctrl set-SR-BR 0
Step 7. Capsule update again from A to B --> Both slot A and B are normal
$ sudo nvbootctrl dump-slots-info
Current version: 35.6.2
Capsule update status: 1
Current bootloader slot: B
Active bootloader slot: B
num_slots: 2
slot: 0, status: normal
slot: 1, status: normal
Please help to review and let us know where you get the issue.
Hi @KevinFFF I think the failure of updating the bootloader has to do with the devices state.
I have two old systems which both were flashed with bootloader form 08/01/2023, 35.4.2 efi version 4.1-33958178.
When I flash that old system on a board and try to update the bootloader, that works flawlessly.
But if i try to update the older systems with the same update file, it gets ignored.
Unfortunately I can’t update the uefi to the debug version, since a reflash obviously would remove all data and make it work.
Can you find any hint about what could be the issue why the update is not starting? The .Cap file is deleted after the reboot.
When mounting manually I had this message:
[ 130.927407] FAT-fs (nvme0n1p12): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Running fsck did not solve the issue tough.
serial.log (115.4 KB)
Okay now I am really confused. I reflashed the board, which was not updateable.
It still can’t be updated, even though it was flashed with the same image as another board, where it works.
I added the DEBUG UEFI, can you see what’s going wrong here?
DebugLog.txt (247.4 KB)
I think I found the reason why the update fails. It works on 3668-300 modules and fails on 3668-301 modules.
What do I need to change in the capsule creation so that the update works with the 301 modules too?
Nevermind, seems to be the capsule payload which is not generated for 301.
Hi,
I have been facing similar issue on Xavier NX board with a multi-spec BUP generated for FAB=301 and FAB=100. Although, the Capsule update can be triggered for FAB=301 board, it cannot for FAB=100.
How did you resolve the issue with different FAB ? Do you need to create a new single-spec BUP with different FAB ? Can’t we do with a multi-spec BUP ?
Do I have to clear scratch-register too ?
@KevinFFF
It seems like the sporadic fail in switching the slot during the bootloader update is not solved by adding all FABs to the capsule.
Can you please have a look any tell me why the CapsuleUpdate did not switch the slot?
serialFailedSlotSwitch.log (79.6 KB)
Obviously the bootloader thinks a partition is missing: bpmp-fw_b
But it exists?
/dev/mmcblk0p7: PARTLABEL=“bpmp-fw_b” PARTUUID=“1258e48b-a7e0-4c05-8997-6b4091307519”
I tried it again and it fails.
What I did:
- Flash 2 boards with massflash with an older version
- update with capsule update
- 301 FAB works
- 300 FAB complains that the bpmp-fw_b partition is missing
What could cause this?
There’re few reasons may cause capsule update not been triggered
- spec does not match
=> runcat /etc/nv_boot_control.confto check your TNSPEC - capsule payload can not be found
=> runefibootmgrto check the boot-order and confirm you put the capsule payload to the correct esp of your boot device - use custom board config but not update board spec
=> add you custom board config in jetson_board_spec.cfg
From the log you shared, it seems bpmp-fw_b can not be found in MB2 boot so that it switches back to slot 0.
Is your current issue specific to FAB 300 device?
If so, have you tried using the following flow to generate the capsule payload for FAB 300 device?
1. Set the device in forced-recovery state can connect USB to host
2. Generate multi-spec BUP
$ sudo ./l4t_generate_soc_bup.sh t19x
3. Generate capsule payload
$ ./generate_capsule/l4t_generate_soc_capsule.sh -i bootloader/payloads_t19x/bl_only_payload -o ./TEGRA_BL.Cap t194
Do you have other boot device like NVMe connected on your board?