Clarification of OTA Update Process

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

As I mention here:

https://forums.developer.nvidia.com/t/agx-xavier-jetson-linux-image-based-ota-update-from-r32-7-x-to-r35-1-fails-black-screen-with-flashing-cursor/239385/11

the update process appears broken. (I have linked it for a reason, please read it)

Can you please clarify whether we can use the dist-upgrade method described in the documentation (here) to update from a minor release (i.e. XX.YY.ZZ), say from r32.5.1 to r32.7.1, then update from r32.7.1 to r35.1.0 via Image Based OTA?

I mean I know the answer, no, but if we can’t do that then can you please:

  1. provide a more concise set of steps to perform the upgrade;
  2. elaborate on what dist-upgrade actually does, if not upgrade the L4T.

It seems as though it basically only updates the apt index, and doesn’t do anything about the underlying system images.

We’ve just run into the issue again when trying to update a jetson that was flashed with JetPack 4.3 and dist-upgrade’d to JetPack 4.5.

If you can point me towards any more documentation I’m happy to read it.

4

Upon repeating the process, it appears that we can reliably follow the dist-upgrade from R32 → R32.5 and then do the Image Based OTA from R32.5 → R32.7.2.

But for some reason the ./tools/l4t_create_default_user.sh script seems to not bypass the oem-config as expected on the new image that is created. The OTA hangs waiting for input to the oem-config stuff.

Any advice on bypassing this and ensuring it is bypassed in the OTA upgrade?

I have run the create default user script in all relevant places with no luck.

@JerryChang

5

Also still waiting on a response on the original question too.

6

Hi khan.schroder-turner,

We are not suggest use Debian Package update and Image-based OTA at same time, that will get unpredictable results. Please use Image-based OTA from r32.5.1 → r32.7.2 → r35.1.
We have workaround to run two times Image-based OTA.

Detail steps:

  1. Generate a OTA package that updates bootloader only from R32.5.1 to R32.7.2 with -b option
    $ sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -b jetson-agx-xavier-devkit R32-5
  2. Apply it on the failed system to update bootloader chain A.
  3. Edit /etc/nv_tegra_release to r32.5.1 before run nv_ota_start.sh
  4. Reboot device and make sure it is booting from chain A after OTA is finished.
    $ sudo nvbootctrl dump-slots-info
    $ sudo nvbootctrl set-active-boot-slot 0
    $ sudo reboot
  5. Edit /etc/nv_tegra_release back to r32.7.2.
  6. Continue OTA from R32.7.2 to R35.1
7

Re-flashing and testing now. Will report back soon.

8

First try results not great.

After updating (with just the bootloader, which i hadnt tried previously), i.e. step 2., it doesnt look like it actually updated.
/etc/nv_tegra_release still showed r35.1.

I proceeded anyway with the following steps, and the R32.7.2 → R35.1.0 ended up in a boot cycle.

Going to try again now, but any thoughts?

9

Actually @carolyuu, now the board cant even be flashed…

[0000.053] W> RATCHET: MB1 binary ratchet value 4 is too la I> MB1 (prd-version: 1.5.1.6-t194-41334769-1740dd39)
[0000.06700.078] I> ATE fuse revision : 0x200
[0000.082] I> Ram repair f> Boot-device: eMMC
[0000.109] I> sdmmc DDR50 mode
[0000.113] E> No bootable slot found
[0000.116] E> LOAD0 from loader.
[0000.128] E> LOADER: Failed to get info for bink 24 failed (err: 0x1d540102)
[0000.145] E> Top caller module: tus dump :
0000000000011111111110111000000000000000000000000000```
10

Hey @carolyuu, it looks like the jetson is properly bricked now. Can you please advise how to fix it?

This is the serial output following a flash to r32.7.2. I tried a flash to r32.5.1 before with similar results.

Writing BMP partition.
[0347.415] I> Writing BMP_b partition.
[0347.439] I> Writing recovery partition.
dtb partition.
[0349.322] I> Writing kernel-bootctrl partition.
[0349.335] I> Writing kernel-bootctrl_b partition.
[0349.349] I> Writing kernel partition.
[0351.082] I> Writing kernel_b partition.
[0352.855] I> Writing kernel-dtb partition.
[0357.101] I> Writing BCT partition.
[0361.556] I> Writing MB1_BCT partition.
62] I> MB1 (prd-version: 1.5.1.9-t194-41334769-73a9b7ef)
[0000.r fuse : 0x0
[0000.085] I> Ram Code : 0x2
[0000.088] I> rst_so] I> Boot-device: eMMC
[0000.109] I> sdmmc DDR50 mode
[0000.113] E> No bootable slot found
[0000.116] E> Lary 0 from loader.
[0000.128] E> LOADER: Failed to get info forTask 24 failed (err: 0x1d540102)
[0000.145] E> Top caller modul01
[0000.154] I> MB1(1.5.1.9-t194-41334769-73a9b7ef) BIT boot status dump :
00000000000111111111101110000000000000000000000000000000000000000000000000000000000000000000000
11

Update - re-flashing twice with r35.1.0 has revived the jetson. Going to flash back to r32.5.1 to try again.

I’m slightly worried about trying the bootloader OTA, I’ll try a full OTA bc I’ve never faced any issues with that before and I dont want to brick the device again. Let me know if this wont work, or whether it will be fine but just slower

Edit: I also took logs if you wanted to see, but I dont think theyll be too helpful.

12

Hi khan.schroder-turner,

Please make sure your BSP package is clean.
Suggest you can prepare clean BSP and start the Image-based OTA.
If follow my steps still fail, please share command line and uart logs for check.

13

Also, your r32.5.2 documentation links to r35.1.0 see here:

Look at the url

14

OK I tried this again, but it doesn’t update the revision in /etc/nv_tegra_release.

Going to continue with the remaining steps now.

I’ve done everything from fresh clean packages.

15

Yep, this definitely doesn’t work. At least with the -b flag.

See serial logs attached for both updates.
ota-3252-3272.cap (67.8 KB)
ota-3272-3510.cap (673.4 KB)

16

Guys, to be honest this is getting ridiculous - especially with the slow response times. Have you actually tested this yourselves?

17

The Jetson is stuck in a never ending boot cycle, I cant get it to go into recovery mode to re-flash.

I can get the bash console, but there’s not much I can do from there.

Please can I get a bit more support here. I see multiple responses per day on other posts, but 1 per day at best with mine.

For context, we’re trying to build commercial products with the Jetsons so we can sell more for you. Its in both our interests to make this work.

18

For anyone who finds themselves unlucky enough to end up in the never ending boot cycle, but can access the bash terminal through the serial console, do the following:

  1. Mount the bootloader stuff:
    mount /dev/mmcblk0p1 /mnt
  2. Check that /mnt/boot/extlinux/extlinux.conf has this stuff:
    root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4
  3. If it does, I cant help you (if it doesnt exist, then create the file - lookup what it should look like). If it doesn’t you’ll need to sed it in. For Xavier, you can do the following:
    sed -i "s|^\([ \t]*APPEND \${cbootargs}\) .*|\1 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0|" /mnt/boot/extlinux/extlinux.conf
    for others, refer to here
  4. You should now be able to boot into recovery mode and re-flash.
19

This definitely doesnt work. You need to do the whole system, just ran through the steps without the -b and it works fine.

I still need to test in some slightly different configurations to be sure.

20

@carolyuu @JerryChang

When I try the OTA from 32.3.1 → 32.7.2, do the bootslot work around, using the patched update scripts from this post, the jetson ends up on the default IP (192.168.55.1) and without an ethernet interface.

I hit this issue on Friday as well while running through a similar update, the difference being that I dist-upgraded to r32.5.2 on friday, and this time I want from fresh flash of r32.3.1 to r32.7.2.

Can you please provide me steps that work? Ideally one’s you’ve tested yourselves.

See attached serial console log
ota-3231-3272.cap (381.3 KB)

21

@WayneWWW Has seen this problem before:

What I don’t understand is I’m following vanilla NVIDIA provided steps using OTA. Can someone please clarify or provide some patch scripts about what the correct update process is.

22

Bump