Issue that OTA works on some Jetsons but not others

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
23

hello alxhoff,

could you please give a side-by-side comparison of those host machines.

24

I only have jp6 and jp6.2 on one of my host computers to experiment with but the attached worked using ota_tools_R36.4.3_aarch64.tbz2 to create ota_payload_package.tar.gz.

According to the docs that package will update the bootloader/uefi and rootfs A/B. I believe you said above that you don’t have uefi secure boot or disk encryption enabled.

I then hacked your one script and hopefully didn’t delete too much of it in the attached files. I don’t have a free agx orin based robot to test but these might be worth a try.

generate_ota_payload.sh.txt (957 Bytes)
updateToJetpack5.1.5.sh.txt (2.0 KB)

25

Yes all of this isn’t a problem. My problem is that for some reason when our Jetsons were manually flashed the bootloaders were not updated and they cannot handle the OTA process where it updates EMMC from the recovery kernel. I have no issues with OTA if the bootloader supports it.

26

My misunderstanding. I thought issue was with some hosts not being able to update the Orins. What I suggested above presumes about 1.4gb free space on Orin and that all hosts can ssh into the Orins. It then runs entirely on the Orin; taking out the network and any host differences from the failure equation.

27

If you haven’t already, use your script logic above to reconfigure
35.4.1 nvidia-l4t-bootloader based on 35.4.1 Release notes pdf

for HOST in "${HOSTS[@]}"; do
    echo "‎[$HOST] checking bootloader version …"
    CURRENT_VER=$(ssh -o BatchMode=yes "${USER}@${HOST}" \
        "sudo nvbootctrl dump-slots-info | awk '/^Current version:/ {print $3}'")

    echo "    current version = $CURRENT_VER"
    if [[ "$CURRENT_VER" == "0.0.1" ]]; then
    sudo DEBIAN_FRONTEND=noninteractive dpkg-reconfigure nvidia-l4t-bootloader
    ## first reboot writes the inactive slot
    sudo reboot
    
# after the board comes back
sudo DEBIAN_FRONTEND=noninteractive dpkg-reconfigure nvidia-l4t-bootloader
sudo reboot            # second reboot rewrites the other slot

Then ota_update from Jetpack 5.1.2=35.4.1 to 5.1.5=35.6.1 on those Orins.

28

hello alxhoff,

may I double check the actual use-case? are you going to update only bootloader?

here’re some clarification,

  1. Image-based OTA update including updating rootfs and updating bootloader. Updating rootfs is before updating the bootloader.
  2. Updating the bootloader is executed in UEFI. Once updating rootfs is finished, the device reboots, and then UEFI updates the bootloader through UEFI capsule update.
  3. Once the UEFI capsule update is finished, the device reboots to the updated chain; otherwise, the device reboots to the original chain.
30

Hi,

so we have a little more information but it seems all a bit random at the same time.

But just to clarify we are doing a full system flash, ie. complete bootchain, kernel + APP.

We have tested maybe 6-7 machines and come to the conclusion that 7th-8th generation intel machines seem to work but newer machine (including AMD) do not work.

I believe that it is somehow related to the USB hubs inside the machines and potentially any power saving features causing it. We have seen twice now that if we boot from a USB HDD (not a live usb but a HDD where the ubuntu system is actually installed to the USB HDD) then flashing the system (+bootloader) results in a non-factory bootloader, ie. the bootloader was updated.

Here are some system dump info from machines that have worked for us and some that have not

Worked:

flash_host_tim_framework_13_11thGen_model-info.txt (524 Bytes)
flash_host_tim_framework_13_11thGen_system_info.txt (21.3 KB)
flash_host_tim_framework_13_11thGen_usb_info.txt (38.3 KB)

flash_host_alex_hoff_x13_thinkpad_system_info.txt (19.1 KB)
flash_host_alex_hoff_x13_thinkpad_usb_info.txt (633 Bytes)

Not Working:

flash_host_tobi_lenovo_ideapad_pro_16_5th_gen_usb_info.txt (101.5 KB)
flash_host_tobi_lenovo_ideapad_pro_16_5th_gen_system_info.txt (19.4 KB)

flash_host_ross_framework_13_usb_info.txt (103.6 KB)
flash_host_ross_framework_13_system_info.txt (20.9 KB)

  • Also an Intel NUC but I have no info from this machine.
31

hello alxhoff,

but.. my host machine with AMD® Ryzen 7 7700x 8-core processor works normally.

32

has this been run on all of the hosts?

sudo ./tools/l4t_flash_prerequisites.sh

Try nohup so that if ssh connection is broken the command will still run.

#!/usr/bin/bash
set -euo pipefail

OTA_SCRIPT=“./nv_ota_start.sh”
OTA_PACKAGE=“ota_payload_package.tar.gz”
LOGFILE=“ota.log”

echo “[INFO] Running OTA with nohup in background…”

nohup “$OTA_SCRIPT” “$OTA_PACKAGE” > “$LOGFILE” 2>&1 &

echo “[INFO] Output will be logged to $LOGFILE”
echo “[INFO] You can safely exit the SSH session.”

33

And if you do

sudo nvbootctrl dump-slots-info do you get a non 0.0.1 version?

All the machine can flash and get the system working but they flash a bootloader that doesn’t support OTA.

34

Thanks for the reply. They normally run in screen for this case but I didn’t know about this script.

35

it may due to your host does not have zlib1g-dev python library installed.
please give it a try to install all related python library before image flashing.

36

Ok thanks, will try this when I get access to a machine that didn’t work for us.

Cheers

37

Are you sure it’s a python library. I can only find the debian package zlib1g-dev.

All the machines (including those not working) have this library installed.

38

Tried this on a known not working machine and we still get the 0.0.1 bootloader version

39

I still wonder if post 25 above could be analogous to your 0.0.1 bootloader problem.

Following is excerpt from Jetson 35.4.1 release notes.pdf

On the Jetson Orin Nano developer kit, if you need to update the QSPI image
on the target to match the SD Card image version, complete the following steps:
1.
When booting into the desktop, to update the slot B bootloader and
reboot, run the following command:
$ sudo dpkg-reconfigure nvidia-l4t-bootloader
It will boot with the bootloader in slot B.
2.
After rebooting into the desktop, to update slot A bootloader and reboot,
run the following command:
$ sudo dpkg-reconfigure nvidia-l4t-bootloader
It will boot with the bootloader in slot A.
3.
Both slots of the bootloader partition on the devkit have now been updated
to the same SD card image version of the SD card image.
This update will be done automatically in a future release.

40

The confusion deepens. We ordered a laptop to our American office that is an identical ThinkPad to one of the known working machines in our German office and when performing the EXACT same steps on an identical Orin AGX (it is all scripted so quite literally identical steps) the Orin comes online with the factory 0.0.1 bootloader, ie. not OTA compatible.

We have also since found that booting from a USB where ubuntu has been directly installed (not a live USB) does not make all unworking machines work, some yes but some no. At this point this entire process is beyond random. We have tried 10+ different machines, running everything from ubuntu 20 through to 24, booting from the host, from a live USB and from a USB SSD native install and the results are super varying and completely random, also with about a 90%+ failure rate.

This is going to become a large issue for us as we have 60+ Orins that we want to flash in the next 2 weeks, as well as all of our new production robots, that are on the other side of the world so they can be updated OTA in the future. Still, at the moment we have not had any luck in that particular office finding hardware that works for flashing the bootloader.

Please help!!!

41

I have tried this perviously. These features don’t work with the 0.0.1 factory bootloader from what I experienced. I also tried capsule updates to update just the bootloader but these fail in a similar style to the OTA updates.

42

@JerryChang were you able to get a non-0.0.1 bootloader? The flashing itself works, ie. the system boots with the configured rootfs, but the bootloader is gatekeeping all the OTA features we need.

43

I might also be wrong in that the bootloader version 0.0.1 is representative of a jetson that cannot be updated OTA but so far it’s that only thing that has really stood out to me.