How to generate version agnostic OTA packages?

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

Board: AGX Orin Industrial
L4T Version: R36.4.3

I am using a custom board

Hello,

I want to generate an OTA image for my board based on my custom rootfs and a pre-built system.img.

Also, I do not want to do an update, I want to have an OTA image that can do a full setup.

I have run the following command in my BSP_ROOT/Linux_for_Tegra directory:

sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh jetson-agx-orin-devkit-industrial R36-4

But It fails with this message:

SUCCESS: cleaned up BUP tmp files prior to payload creation
FAILURE: no payload made for config “boardid=3701;fab=300;boardsku=0008;boardrev=;fuselevel_s=1;chiprev=;chipsku=00:00:00:90;board=jetson-agx-orin-devkit-industrial;rootdev=internal;signed_img_dir=images-R36-ToT”
SUCCESS: cleaned up BUP tmp files after payload creation
Failed to run ROOTFS_AB= ROOTFS_ENC= /tmp/tmp.wWDrwuClwO/l4t_generate_soc_bup.sh -f /tmp/board_spec_file -e tmp_board_spec -b jetson-agx-orin-devkit-industrial t23x

More log here (I cannot paste the full log)

./tegraflash.py  --bl uefi_jetson_minimal_with_dtb.bin  --odmdata gbe-uphy-config-22,hsstp-lane-map-3,nvhs-uphy-config-0,hsio-uphy-config-0,gbe0-enable-10g  --overlay_dtb L4TConfiguration.dtbo,tegra234-p3737-0000+p3701-0000-dynamic.dtbo,tegra234-carveouts.dtbo,tegra-optee.dtbo,tegra234-p3737-camera-dual-imx274-overlay.dtbo,tegra234-p3737-camera-e3331-overlay.dtbo,tegra234-p3737-camera-e3333-overlay.dtbo,tegra234-p3737-camera-imx185-overlay.dtbo,  --bldtb tegra234-p3737-0000+p3701-0008-nv.dtb --applet mb1_t234_prod.bin --cmd "sign"  --cfg flash.xml --chip "0x23" --concat_cpubl_bldtb --cpubl uefi_jetson.bin --cpubl_rcm uefi_jetson_minimal.bin --minratchet_config tegra234-mb1-bct-ratchet-p3701-0000.dts --device_config tegra234-mb1-bct-device-p3701-0000.dts --misc_cold_boot_config tegra234-mb1-bct-misc-p3701-0008.dts --misc_config tegra234-mb1-bct-misc-p3701-0008-flash.dts --pinmux_config tegra234-mb1-bct-pinmux-p3701-0000-a04.dtsi --gpioint_config tegra234-mb1-bct-gpioint-p3701-0000.dts --pmic_config tegra234-mb1-bct-pmic-p3701-0008.dts --pmc_config tegra234-mb1-bct-padvoltage-p3701-0000-a04.dtsi --deviceprod_config tegra234-mb1-bct-cprod-p3701-0000.dts --prod_config tegra234-mb1-bct-prod-p3701-0000.dts --scr_config tegra234-mb2-bct-scr-p3701-0000.dts --wb0sdram_config tegra234-p3701-0008-wb0sdram-l4t.dts --br_cmd_config tegra234-mb1-bct-reset-p3701-0000.dts --uphy tegra234-mb1-bct-uphylane-si.dtsi --dev_params tegra234-br-bct-p3701-0000.dts,tegra234-br-bct_b-p3701-0000.dts --mb2bct_cfg tegra234-mb2-bct-misc-p3701-0008.dts  --bins "psc_fw pscfw_t234_prod.bin; mts_mce mce_flash_o10_cr_prod.bin; tsec_fw tsec_t234.bin; mb2_applet applet_t234.bin; mb2_bootloader mb2_t234.bin; xusb_fw xusb_t234_prod.bin; pva_fw nvpva_020.fw; dce_fw display-t234-dce.bin; nvdec nvdec_t234_prod.fw; bpmp_fw bpmp_t234-TE992M-A1_prod.bin; fsi_fw fsi-lk.bin; bpmp_fw_dtb tegra234-bpmp-3701-0008-3737-0000.dtb; rce_fw camera-rtcpu-t234-rce.img; ape_fw adsp-fw.bin; spe_fw spe_t234.bin; tos tos-optee_t234.img; eks eks_t234.img"  --sdram_config tegra234-p3701-0008-sdram-l4t.dts  --bct_backup  --boot_chain A  --ramcode 4  --ramcode 4 
saving flash command in /tmp/tmp.wWDrwuClwO/bootloader/flashcmd.txt
saving Windows flash command to /tmp/tmp.wWDrwuClwO/bootloader/flash_win.bat
*** Sign and generate BUP... *** 
Signing with zerosbk ...
/tmp/tmp.wWDrwuClwO/bootloader/l4t_bup_gen.func: line 169: /tmp/tmp.wWDrwuClwO/bootloader/rollback/rollback_parser.py: cannot execute: required file not found
###############################################################################
# L4T BSP Information:
# R36 , REVISION: 4.3
# User release: 0.0
###############################################################################
Board ID(3701) version(300) sku(0008) revision()
Preset RAMCODE is 4
Chip SKU(00:00:00:90) ramcode(4) fuselevel(fuselevel_production) board_FAB(300)
reusing existing /tmp/tmp.wWDrwuClwO/bootloader/badpage.bin
16+0 records in
16+0 records out
8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0001748 s, 46.9 MB/s
Generated UUID 7e647648-49db-43be-9ed2-5150432cedd8 for mounting root APP partition.
copying bctfile(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-p3701-0008-sdram-l4t.dts)... done.
copying minratchet_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-ratchet-p3701-0000.dts)... done.
copying device_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-device-p3701-0000.dts)... done.
copying misc_cold_boot_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-misc-p3701-0008.dts)... done.
copying misc_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-misc-p3701-0008-flash.dts)... done.
copying pinmux_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-pinmux-p3701-0000-a04.dtsi)... done.
copying gpioint_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-gpioint-p3701-0000.dts)... done.
copying pmic_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-pmic-p3701-0008.dts)... done.
copying pmc_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-padvoltage-p3701-0000-a04.dtsi)... done.
copying deviceprod_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-cprod-p3701-0000.dts)... done.
copying prod_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-prod-p3701-0000.dts)... done.
copying scr_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb2-bct-scr-p3701-0000.dts)... done.
copying wb0sdram(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-p3701-0008-wb0sdram-l4t.dts)... done.
copying bootrom_config(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb1-bct-reset-p3701-0000.dts)... done.
Existing uphylane_config(/tmp/tmp.wWDrwuClwO/bootloader/tegra234-mb1-bct-uphylane-si.dtsi) reused.
copying dev_params(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-br-bct-p3701-0000.dts)... done.
copying dev_params_b(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-br-bct_b-p3701-0000.dts)... done.
copying mb2bct_cfg(/tmp/tmp.wWDrwuClwO/bootloader/generic/BCT/tegra234-mb2-bct-misc-p3701-0008.dts)... done.
Existing pscfwfile(/tmp/tmp.wWDrwuClwO/bootloader/pscfw_t234_prod.bin) reused.
Existing pscbl1file(/tmp/tmp.wWDrwuClwO/bootloader/psc_bl1_t234_prod.bin) reused.
Existing mtsmcefile(/tmp/tmp.wWDrwuClwO/bootloader/mce_flash_o10_cr_prod.bin) reused.
Existing tscfwfile(/tmp/tmp.wWDrwuClwO/bootloader/tsec_t234.bin) reused.
Existing mb2applet(/tmp/tmp.wWDrwuClwO/bootloader/applet_t234.bin) reused.
Existing bootloader(/tmp/tmp.wWDrwuClwO/bootloader/mb2_t234.bin) reused.
copying initrd(/tmp/tmp.wWDrwuClwO/bootloader/l4t_initrd.img)... done.
bl is uefi
Making Boot image... done.
Not signing of boot.img
Existing sosfile(/tmp/tmp.wWDrwuClwO/bootloader/mb1_t234_prod.bin) reused.
Existing tegraboot(/tmp/tmp.wWDrwuClwO/bootloader/mb2_t234.bin) reused.
Existing cpu_bootloader(/tmp/tmp.wWDrwuClwO/bootloader/mb2_t234.bin) reused.
Existing mb2blfile(/tmp/tmp.wWDrwuClwO/bootloader/mb2_t234.bin) reused.
Existing xusbfile(/tmp/tmp.wWDrwuClwO/bootloader/xusb_t234_prod.bin) reused.
Existing pvafile(/tmp/tmp.wWDrwuClwO/bootloader/nvpva_020.fw) reused.
Existing dcefile(/tmp/tmp.wWDrwuClwO/bootloader/display-t234-dce.bin) reused.
Existing nvdecfile(/tmp/tmp.wWDrwuClwO/bootloader/nvdec_t234_prod.fw) reused.
Existing psc_rf(/tmp/tmp.wWDrwuClwO/bootloader/psc_rf_t234_prod.bin) reused.
Existing mb2_rf(/tmp/tmp.wWDrwuClwO/bootloader/mb2rf_t234.bin) reused.
Existing mb1file(/tmp/tmp.wWDrwuClwO/bootloader/mb1_t234_prod.bin) reused.
Existing bpffile(/tmp/tmp.wWDrwuClwO/bootloader/bpmp_t234-TE992M-A1_prod.bin) reused.
Existing fsifwfile(/tmp/tmp.wWDrwuClwO/bootloader/fsi-lk.bin) reused.
copying bpfdtbfile(/tmp/tmp.wWDrwuClwO/bootloader/generic/tegra234-bpmp-3701-0008-3737-0000.dtb)... done.
Existing camerafw(/tmp/tmp.wWDrwuClwO/bootloader/camera-rtcpu-t234-rce.img) reused.
Existing apefile(/tmp/tmp.wWDrwuClwO/bootloader/adsp-fw.bin) reused.
Existing spefile(/tmp/tmp.wWDrwuClwO/bootloader/spe_t234.bin) reused.
Existing badpagefile(/tmp/tmp.wWDrwuClwO/bootloader/badpage.bin) reused.
Existing wb0boot(/tmp/tmp.wWDrwuClwO/bootloader/sc7_t234_prod.bin) reused.
Existing tosfile(/tmp/tmp.wWDrwuClwO/bootloader/tos-optee_t234.img) reused.
Existing eksfile(/tmp/tmp.wWDrwuClwO/bootloader/eks_t234.img) reused.
copying dtbfile(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra234-p3737-0000+p3701-0008-nv.dtb)... done.
Copying nv_boot_control.conf to rootfs
Skip generating system.img
Not signing of kernel-dtb
Existing tbcfile(/tmp/tmp.wWDrwuClwO/bootloader/uefi_jetson.bin) reused.
Existing rcm_tbcfile(/tmp/tmp.wWDrwuClwO/bootloader/uefi_jetson_minimal.bin) reused.
copying tbcdtbfile(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra234-p3737-0000+p3701-0008-nv.dtb)... done.
copying cfgfile(/tmp/tmp.wWDrwuClwO/bootloader/generic/cfg/flash_t234_qspi_sdmmc_industrial.xml) to flash.xml... done.
Existing flashapp(/tmp/tmp.wWDrwuClwO/bootloader/tegraflash.py) reused.
copying overlay_dtb(/tmp/tmp.wWDrwuClwO/kernel/dtb/L4TConfiguration.dtbo)... done.
copying overlay_dtb(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra234-p3737-0000+p3701-0000-dynamic.dtbo)... done.
copying overlay_dtb(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra234-carveouts.dtbo)... done.
copying overlay_dtb(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra-optee.dtbo)... done.
copying overlay_dtb(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra234-p3737-camera-dual-imx274-overlay.dtbo)... done.
copying overlay_dtb(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra234-p3737-camera-e3331-overlay.dtbo)... done.
copying overlay_dtb(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra234-p3737-camera-e3333-overlay.dtbo)... done.
copying overlay_dtb(/tmp/tmp.wWDrwuClwO/kernel/dtb/tegra234-p3737-camera-imx185-overlay.dtbo)... done.
/tmp/tmp.wWDrwuClwO/bootloader/L4TConfiguration_updated.dts: Warning (unit_address_vs_reg): Node /fragment@0 has a unit name, but no reg property
/tmp/tmp.wWDrwuClwO/bootloader/L4TConfiguration.dtbo: Warning (unit_address_vs_reg): Node /fragment@0 has a unit name, but no reg property
./tegraflash.py  --bl uefi_jetson_minimal_with_dtb.bin  --odmdata gbe-uphy-config-22,hsstp-lane-map-3,nvhs-uphy-config-0,hsio-uphy-config-0,gbe0-enable-10g  --overlay_dtb L4TConfiguration.dtbo,tegra234-p3737-0000+p3701-0000-dynamic.dtbo,tegra234-carveouts.dtbo,tegra-optee.dtbo,tegra234-p3737-camera-dual-imx274-overlay.dtbo,tegra234-p3737-camera-e3331-overlay.dtbo,tegra234-p3737-camera-e3333-overlay.dtbo,tegra234-p3737-camera-imx185-overlay.dtbo,  --bldtb tegra234-p3737-0000+p3701-0008-nv.dtb --applet mb1_t234_prod.bin --cmd "sign"  --cfg flash.xml --chip "0x23" --concat_cpubl_bldtb --cpubl uefi_jetson.bin --cpubl_rcm uefi_jetson_minimal.bin --minratchet_config tegra234-mb1-bct-ratchet-p3701-0000.dts --device_config tegra234-mb1-bct-device-p3701-0000.dts --misc_cold_boot_config tegra234-mb1-bct-misc-p3701-0008.dts --misc_config tegra234-mb1-bct-misc-p3701-0008-flash.dts --pinmux_config tegra234-mb1-bct-pinmux-p3701-0000-a04.dtsi --gpioint_config tegra234-mb1-bct-gpioint-p3701-0000.dts --pmic_config tegra234-mb1-bct-pmic-p3701-0008.dts --pmc_config tegra234-mb1-bct-padvoltage-p3701-0000-a04.dtsi --deviceprod_config tegra234-mb1-bct-cprod-p3701-0000.dts --prod_config tegra234-mb1-bct-prod-p3701-0000.dts --scr_config tegra234-mb2-bct-scr-p3701-0000.dts --wb0sdram_config tegra234-p3701-0008-wb0sdram-l4t.dts --br_cmd_config tegra234-mb1-bct-reset-p3701-0000.dts --uphy tegra234-mb1-bct-uphylane-si.dtsi --dev_params tegra234-br-bct-p3701-0000.dts,tegra234-br-bct_b-p3701-0000.dts --mb2bct_cfg tegra234-mb2-bct-misc-p3701-0008.dts  --bins "psc_fw pscfw_t234_prod.bin; mts_mce mce_flash_o10_cr_prod.bin; tsec_fw tsec_t234.bin; mb2_applet applet_t234.bin; mb2_bootloader mb2_t234.bin; xusb_fw xusb_t234_prod.bin; pva_fw nvpva_020.fw; dce_fw display-t234-dce.bin; nvdec nvdec_t234_prod.fw; bpmp_fw bpmp_t234-TE992M-A1_prod.bin; fsi_fw fsi-lk.bin; bpmp_fw_dtb tegra234-bpmp-3701-0008-3737-0000.dtb; rce_fw camera-rtcpu-t234-rce.img; ape_fw adsp-fw.bin; spe_fw spe_t234.bin; tos tos-optee_t234.img; eks eks_t234.img"  --sdram_config tegra234-p3701-0008-sdram-l4t.dts  --bct_backup  --boot_chain A  --ramcode 4 
saving flash command in /tmp/tmp.wWDrwuClwO/bootloader/flashcmd.txt
saving Windows flash command to /tmp/tmp.wWDrwuClwO/bootloader/flash_win.bat
*** Sign and generate BUP... *** 
t23x payload generation complete with 1 failure(s)

SUCCESS: cleaned up BUP tmp files prior to payload creation
FAILURE: no payload made for config "boardid=3701;fab=300;boardsku=0008;boardrev=;fuselevel_s=1;chiprev=;chipsku=00:00:00:90;board=jetson-agx-orin-devkit-industrial;rootdev=internal;signed_img_dir=images-R36-ToT"
SUCCESS: cleaned up BUP tmp files after payload creation
Failed to run ROOTFS_AB= ROOTFS_ENC=  /tmp/tmp.wWDrwuClwO/l4t_generate_soc_bup.sh -f /tmp/board_spec_file -e tmp_board_spec -b jetson-agx-orin-devkit-industrial t23x

Do you have any idea what could be wrong ?

Thanks !

4

Figured out what was this error:

/tmp/tmp.wWDrwuClwO/bootloader/l4t_bup_gen.func: line 169: /tmp/tmp.wWDrwuClwO/bootloader/rollback/rollback_parser.py: cannot execute: required file not found

Shebang of bootloader/rollback/rollback_parser.py was usr/bin/python which didn’t exist. I installed python-is-python3 to solve the issue.

My question about the version agnostic OTA images still stands though.

Ideally I would like to re-partition and re-write everything

6

hello user27558,

I do not understand your question, please help me to understand this clearly, or, please share an example for reference.

7

I understand that OTA updates are supported only from version 35.5.0 onward, and that the current system performs version-specific upgrades (for example, from version A to B).

I’d like to know if it would be possible to create one OTA image that works on any OTA-capable device starting from version 35.5.0. The idea is to have a single image that performs a complete reinstall of the system rather than an incremental update.

Ideally, this process would fully reset the device, removing all existing data and configuration so it starts fresh with the new software. We plan to use the existing A/B partition system for rollback and recovery in case something goes wrong during the update.

Could you please confirm if such a universal full-reinstall OTA image is supported? If not, I’d appreciate any guidance on what would be required to make it possible.

Thanks a lot for your help,

8

hello user27558,

An universal full-reinstall OTA image is not supported.
currently, it’s needed to configure BASE_BSP/TARGET_BSP for creating OTA payload, it’s version-specific upgrades.

9

OTA upgrades, are able to re-partition and write to partitions, so it should be possible right ?

The current method of having to build OTA images for each BSP version does not scale, we need to find a solution to update our boards.

10

Hello @user27558,

How is everything going?

I found this topic very interesting.

So essentially you just need a way to remotely “factory reset” your devices right ?

If I am understand that correctly, then you are looking for a way to build OTA payloads that will only “update” the rootfs, where the update is simply going back to what was flashed from the factory.

The problem hover, I think you found not in building the payload itself, but having to build a payload for every BSP version that is flashed to a device in the field right ?

best regards,
Andrew
Embedded Software Engineer at ProventusNova

11

Hey @proventusnova

So essentially you just need a way to remotely “factory reset” your devices right ?

You can put it this way yes.

If I am understand that correctly, then you are looking for a way to build OTA payloads that will only “update” the rootfs, where the update is simply going back to what was flashed from the factory.

Not only the rootfs, ideally I would like to re-partition averything and rewrite each partition, just like a full flash would do it.

The problem hover, I think you found not in building the payload itself, but having to build a payload for every BSP version that is flashed to a device in the field right ?

Correct, it does not scale. We have different boards, running different L4T versions, we cannot maintain so many OTA images.

12

@user27558 ,

Got it, yeah.
Is a tough problem to solve.

Given the limitation of the OTA Update system, which does not support full-reinstall at the time. One idea that comes to mind is leveraging the A/B support.

I would try having one of the two partitions always flashed with the same JP for all devices. So you can use that partition only for maintenance and updates. So regardless of the JP you run on the other partition, you can always switch to that one and start the update procedure.

Although I know is not going to be friendly with the devices that are already deployed in the field. You would need to go through and custom update process for each one of them. But once you get your whole fleet flashed with that maintenance partition scheme… next updates would not be an issue since you would be deploying the same OTA payload package.

Just an idea, don’t really know if it will work for your current circumstances.

best regards,
Andrew
Embedded Software Engineer at ProventusNova

13

@proventusnova, Thanks, I will look into that.

@JerryChang , Can you comment ? Do you have any suggestions?

14

hello user27558,

as mentioned, currently, it’s needed to configure BASE_BSP/TARGET_BSP for creating OTA payload, it’s version-specific upgrades.

15

Hey @JerryChang, thanks for your answer.

But OTA updates are able to repartition, and write to partitions (it reboots to forced recovery).
What prevents it to re-partition everything and re-write all partitions ?

Is this feature planned in the future ?

16

hello user27558,

it still depends-on your base BSP version.
for instance, if you based a version earlier than r35.5.0, you must moving to r35.5.0 and then moving forward.