OTA Update r35.4.1 on an external device nvme0n1p1 failed to update NVMe

[adit_bhrgv]

Hi,

FYI:
Using Jetson Linux R35.4.1 release
Jetson Xavier NX custom platform

I wanted to create a bootable USB which can perform the OTA update on the external drive such as nvme0n1p1 . For this to happen, I did follow below steps:

1. Payload Generation

# Setting BSP Environmental variables
$ export BASE_BSP=$pwd/base/Linux_for_Tegra
$ export TARGET_BSP=$pwd/target/Linux_for_Tegra

# Install OTA tools inside TARGET code repository.
$ cd ${TARGET_BSP}/../
$ wget https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v4.1/release/ota_tools_r35.4.1_aarch64.tbz2
$ sudo tar xpf ota_tools_r35.4.1_aarch64.tbz2

#Generate OTA payload package(Including Bootloader and Rootfs) for nvme device.
$ cd ${TARGET_BSP}
$ export TARGET_JETSON_DEVICE=jetson-xavier-nx-devkit-emmc
$ export BASE_BSP_VERSION=R35-4
$ sudo -E ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh --external-device nvme0n1 -S 40GiB ${TARGET_JETSON_DEVICE} ${BASE_BSP_VERSION}

2. Formatted a USB stick without creating any partitions (full 128 GB as free-space)

#For example, the 128 Gb USB is identified as /dev/sdh 
$lsblk 
sdh           8:112  1 119,3G  0 disk  

3. Prepare bootable USB for OTA update using l4t_initrd_flash.sh

$ cd ${pwd}/target/Linux_For_tegra
$ mkdir /rootfs/ota
$ cd /rootfs/ota
$ wget ${OTA_SERVER}/payload_file
$ wget ${OTA_SERVER}/OTA_Tools_file
$ cd ../../
$ ./tools/kernel_flash/l4t_initrd_flash.sh --direct sdh --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_nvme.xml -S 40GiB jetson-xavier-nx-devkit-emmc external

# On successful run on this command, bootable USB created with the rootfs from Linux_For_Tegra folder.

4. Connect the Bootable USB to target jetson and select “Boot from USB” from Boot Menu

# Run the OTA update using payload file
$ /ota/Linux_for_Tegra/tools/ota_tools/version_upgrade/nv_ota_start.sh /ota/ota_payload_package.tar.gz

#Runs successfully without any errors

5. Remove the USB, Reboot the jetson and on bootup select “Boot from NVME”. Then it didn’t trigger the OTA update and the monitor just shows a blank screen with a cursor and then the display goes off.

• Could you please tell me is the method to update the NVME using USB I described above is correct or am I doing something wrong ?

• Could you please tell me on when it triggers the update on bootup ? Can I see some logs that ota update started with progress bar etc. ? My understanding is it will replace the rootfs folder inside /ota folder into /nvme/rootfs folder

Thanks !

[KevinFFF]

Hi adit_bhrgv,

@adit_bhrgv

Could you please tell me is the method to update the NVME using USB I described above is correct or am I doing something wrong ?

You should put OTA update payload to the NVMe drive.

@adit_bhrgv

Could you please tell me on when it triggers the update on bootup ? Can I see some logs that ota update started with progress bar etc. ? My understanding is it will replace the rootfs folder inside /ota folder into /nvme/rootfs folder

At least you would see the update progress bar in UEFI during boot up.

Please share the full serial console to check what’s happening inside.

[adit_bhrgv]

Before I can provide you the faulty serial logs, I tried to first setup the minicom on a working image on NVME but there I found another issue that the boot logs are not getting printed on minicom.

I connected the jetson NX COM port to the Serial-USB connector to the host PC.
I setup the minicom on the device /dev/ttyACM2 but I couldn’t see any boot logs. Although, after sometime, I can see the login screen where I can give the username and password.

Could you please tell why can’t I see the boot logs on the minicom ?

I checked this blog: Not getting Boot logs in console - #3 by vinay2010singh

I don’t have “quiet” in any of the below places on the Jetson NX :

• /boot/extlinux/extlinux.conf

• cat /proc/cmdline

• cat /proc/device-tree/chosen/bootargs

[adit_bhrgv]

ota_logs.txt (3.2 KB)

I also attached the ota_logs.txt here.

[adit_bhrgv]

Note that my / is mounted on USB stick when I do ./nv_ota_start.sh /ota/payload.tar.gz
I noticed that in my /boot/extlinux/extlinux.conf file, another entry with root=PARTUUID is appended . This is from the USB .
Is this correct behaviour ?
I attach the conf file for reference.
extlinux.conf.txt (1.2 KB)

[KevinFFF]

@adit_bhrgv

Could you please tell why can’t I see the boot logs on the minicom ?

For the Xavier NX devkit, you could refer to the following instruction to capture serial console log.
Jetson Nano & NX Style - Serial Debug Console - JetsonHacks
Since you are using custom carrier board, please check with your HW team or vendor for where’s the debug UART pin.

@adit_bhrgv

Note that my / is mounted on USB stick when I do ./nv_ota_start.sh /ota/payload.tar.gz

Do you want to perform OTA update for external NVMe or USB?

[adit_bhrgv]

Ok i will check with HW vendor team. Thank you.

I want to perform OTA update for external NVMe through USB.

You can see the use case here :

Use OTA Payload to reinstall Jetson System

Hello, I have a question regarding updating or flashing systems that are at the customer or in the field. Our service would like to be able to reinstall a Jetson in the field as well. To put the Jetson into recovery mode and always have a notebook with ubuntu available is unfortunately not a solution for our service. The desired solution would be a USB stick from which you can boot and which performs a new installation. We are having a custom board with Xavier NX, the rootfs is on a nvme S…

[KevinFFF]

@adit_bhrgv

I want to perform OTA update for external NVMe through USB.

You should put the OTA payload to the external device which you would like to perform update.
In your case, it seems NVMe drive.

[adit_bhrgv]

@KevinFFF

he OTA payload to the external device which you would like to perform update

Yes, the NVME has to be OTA updated.
In this case, once I boot from USB(/dev/sda1) and the rootfs is currently on USB.

You mean I first create a mount point for /dev/nvme0n1p1 to lets say (/mnt), place the OTA payload in /mnt and then do as below :
/ota/Linux_for_Tegra/tools/ota_tools/version_upgrade/nv_ota_start.sh /mnt/ota_payload_package.tar.gz. as the OTA payload was generated for NVME with options --external-device nvme0n1 -S 40GiB

[KevinFFF]

@adit_bhrgv

You mean I first create a mount point for /dev/nvme0n1p1 to lets say (/mnt), place the OTA payload in /mnt and then do as below :
/ota/Linux_for_Tegra/tools/ota_tools/version_upgrade/nv_ota_start.sh /mnt/ota_payload_package.tar.gz. as the OTA payload was generated for NVME with options --external-device nvme0n1 -S 40GiB

Yes, or just remove USB and boot from NVMe to put OTA payload in rootfs.
nv_ota_start.sh is used to check and prepare for the OTA update. The actual OTA update process happens after you run reboot command.

[adit_bhrgv]

Note: There is no USB attached.

I just moved OTA payload tar to NVME /ota/ & then did ran ./nv_ota_start.sh /ota/ota_payload_package.tar.gz . It performed well without errors.
Then I executed “sudo reboot”. After reboot, I selected NVME as boot-device but I can’t see any OTA update progress bar. There is a black screen on monitor with a cursor on the top left corner and then eventually the HDMI power went off.

Could you please tell what could be the issue ?

Thanks

[adit_bhrgv]

Hi,

it seems the OTA update on NVME is working.

• Although, I do see a black monitor with cursor and monitor power going off after sudo reboot, this time I waited a bit (~8-10 mins). The monitor power came to life again and when booted from NVME, it do see the OTA payload content in rootfs. However, the hostname got changed to localhost.localdomain instead of hostname I set during creating rootfs which is used to generate the payload. Do you know why this could be the case ?
  Also, Here, I am not sure, how can I monitor the progress of OTA update when the monitor goes powerless. Probably serial console by asking our vendor ?

• Coming again to the our use-case ( Use OTA Payload to reinstall Jetson System )I mentioned earlier in comments. Can I use the bootable USB to OTA update the NVME ? Basically what we want is to reinstall the jetson via USB through OTA payload without going through “Setting the jetson into recovery”. Which means even if the NVME is not booting up earlier, we should be able to install a new system on NVME via USB. Is this concept feasible ?

• I noticed one more thing, once the OTA update is finished !

Before OTA update: , /ota folder had 1 file a.txt
/ota/
└── a.txt

0 directories, 1 file

After OTA update: , /ota folder had 2 files a.txt and b.txt
/ota/
├── a.txt
└── b.txt

0 directories, 2 files

Should the file a.txt persist after OTA update ? I expected just b.txt file (coming from OTA payload ) will be present in /ota folder. Is this behaviour normal ?

Thanks again for your help !

[KevinFFF]

@adit_bhrgv

Although, I do see a black monitor with cursor and monitor power going off after sudo reboot, this time I waited a bit (~8-10 mins). The monitor power came to life again and when booted from NVME, it do see the OTA payload content in rootfs. However, the hostname got changed to localhost.localdomain instead of hostname I set during creating rootfs which is used to generate the payload. Do you know why this could be the case ?

I’m not clear about this behavior w/o checking serial console log. Maybe it was performing OTA update?

@adit_bhrgv

Here, I am not sure, how can I monitor the progress of OTA update when the monitor goes powerless. Probably serial console by asking our vendor ?

Correct! Please get serial console log to check what is happening inside.

@adit_bhrgv

Can I use the bootable USB to OTA update the NVME ? Basically what we want is to reinstall the jetson via USB through OTA payload without going through “Setting the jetson into recovery”. Which means even if the NVME is not booting up earlier, we should be able to install a new system on NVME via USB. Is this concept feasible ?

You could put OTA payload to USB first and connect it to your board, but you should still need to move the OTA payload from USB to the /ota in your NVMe drive.

What is the content of a.txt and b.txt?

[adit_bhrgv]

a.txt → ota_payload.tar.gz ( which I put inside /ota before starting OTA Update)

b.txt → extracted OTA tools ( Linux_for_Tegra) folder which was present in rootfs/ota folder from which the payload was created. I could empty this /ota folder too as I download and extracted OTA tools at /home/user folder.

[adit_bhrgv]

We have custom jetson board with secure boot and disk encryption enabled. Currently I tried the OTA update on unfused board and without disk encryption.

According to below post, it seems OTA update cannot run on encrypted disks. Could you please confirm this ? If yes, how can we run OTA update then ? Is there any alternatives ? eg: “dd” all partition images to nvme via flash.sh ?

OTA upgrade on devices that have secure boot and disk encryption

hello HusamAlqaza, we had some internal discussion… there’s no OTA with disk encryption. for example, JP5.1.2 → JP5.1.3 with disk encryption is not supported. hence, only the devices with PKC+SBK are supported with OTA upgrade at the moment.

[KevinFFF]

Image-based OTA update would be supported from the next R35 release.
You should enable disk-encryption on your board now, and you could perform OTA update to next R35 release.