Reflashing error after OTA

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
,
1

I have followed the guide for image-based OTA here: Software Packages and the Update Mechanism — Jetson Linux Developer Guide documentation

I was able to successfully update the rootfs and bootloader from jetson 35.3.1 to jetson 35.4.1 on an Orin dev kit.

However, when trying to reflash with jetson 35.4.1, I am getting the following error:

###############################################################################
# L4T BSP Information:
# R35 , REVISION: 4.1
# User release: 0.0
###############################################################################
# Target Board Information:
# Name: jetson-agx-orin-devkit, Board Family: t186ref, SoC: Tegra 234, 
# OpMode: production, Boot Authentication: NS, 
# Disk encryption: disabled ,
###############################################################################
copying emc_fuse_dev_params(/home/ubuntu/Documents/latest/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-br-bct-diag-boot.dts)... done.
copying device_config(/home/ubuntu/Documents/latest/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-mb1-bct-device-p3701-0000.dts)... done.
copying misc_config(/home/ubuntu/Documents/latest/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-mb1-bct-misc-p3701-0000.dts)... done.
./tegraflash.py --chip 0x23 --applet "/home/ubuntu/Documents/latest/Linux_for_Tegra/bootloader/mb1_t234_prod.bin" --skipuid --cfg readinfo_t234_min_prod.xml --dev_params tegra234-br-bct-diag-boot.dts --device_config tegra234-mb1-bct-device-p3701-0000.dts --misc_config tegra234-mb1-bct-misc-p3701-0000.dts --bins "mb2_applet applet_t234.bin" --cmd "dump eeprom cvm cvm.bin; dump custinfo custinfo_out.bin; reboot recovery" 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0215 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.0226 ] File rcm_state open failed
[   0.0226 ] ERROR: failed to read rcm_state
[   0.0226 ] 
[   0.0245 ] tegrasign_v3.py --key None --getmode mode.txt
[   0.0247 ] Assuming zero filled SBK key
[   0.0231 ] Pre-processing config: tegra234-mb1-bct-device-p3701-0000.dts
[   0.0300 ] Pre-processing config: tegra234-mb1-bct-misc-p3701-0000.dts
[   0.0418 ] Parsing partition layout
[   0.0433 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.0449 ] Kernel DTB used: None
[   0.0449 ] WARNING: dce base dtb is not provided

[   0.0449 ] Parsing partition layout
[   0.0458 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.0472 ] Creating list of images to be signed
[   0.0482 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --list images_list.xml zerosbk
[   0.0492 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.0515 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.0523 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.0583 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.0591 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.0636 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.0651 ] adding BCH for mb2_t234_aligned.bin
[   0.0675 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.0789 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.0808 ] adding BCH for mb2_t234_aligned.bin
[   0.0956 ] Filling MB1 storage info
[   0.0957 ] Parsing dev params for multi chains
[   0.1040 ] Generating br-bct
[   0.1051 ] Updating dev and MSS params in BR BCT
[   0.1052 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.1085 ] Updating bl info
[   0.1100 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin
[   0.1112 ] WARNING: boot chain is not completed. set to 0
[   0.1126 ] Generating signatures
[   0.1145 ] tegrasign_v3.py --key None --list images_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1147 ] Assuming zero filled SBK key
[   0.1325 ] Warning: pub_key.key is not found
[   0.1311 ] Parsing dev params for multi chains
[   0.1312 ] Generating br-bct
[   0.1324 ] Updating dev and MSS params in BR BCT
[   0.1325 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.1355 ] Updating bl info
[   0.1390 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.1406 ] WARNING: boot chain is not completed. set to 0
[   0.1433 ] Get Signed section of bct
[   0.1467 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --listbct bct_list.xml
[   0.1492 ] Signing BCT
[   0.1511 ] tegrasign_v3.py --key None --list bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1513 ] Assuming zero filled SBK key
[   0.1546 ] Sha saved in br_bct_BR.sha
[   0.1549 ] Warning: pub_key.key is not found
[   0.1532 ] Updating BCT with signature
[   0.1540 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesig bct_list_signed.xml
[   0.1548 ] Offset :4608 Len :3584
[   0.1557 ] Generating SHA2 Hash
[   0.1576 ] tegrasign_v3.py --key None --list bct_list.xml --sha sha512
[   0.1578 ] Assuming zero filled SBK key
[   0.1579 ] Assuming zero filled SBK key
[   0.1621 ] Sha saved in br_bct_BR.sha
[   0.1610 ] Updating BCT with SHA2 Hash
[   0.1622 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesha bct_list_signed.xml
[   0.1631 ] Offset :4608 Len :3584
[   0.1638 ] Offset :68 Len :8124
[   0.1638 ] 
[   0.1639 ] Generating coldboot mb1-bct
[   0.1649 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct.cfg --misc tegra234-mb1-bct-misc-p3701-0000_cpp.dtb --device tegra234-mb1-bct-device-p3701-0000_cpp.dtb
[   0.1658 ] MB1-BCT version: 0.10
[   0.1660 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED1/ is not supported
[   0.1662 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED1/ is not supported
[   0.1664 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED1/ is not supported
[   0.1666 ] ERROR: /misc/tsc_controls/tsc_locking_config is not supported
[   0.1669 ] ERROR: /misc/tsc_controls/tsc_locking_diff_configuration is not supported
[   0.1672 ] ERROR: /misc/tsc_controls/tsc_locking_ref_frequency_configuration is not supported
[   0.1678 ] ERROR: /misc/tsc_controls/tsc_locking_control is not supported
[   0.1679 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_configuration is not supported
[   0.1679 ] ERROR: /misc/tsc_controls/tsc_locking_fast_adjust_configuration is not supported
[   0.1679 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_delta_control is not supported
[   0.1679 ] ERROR: /misc/tsc_controls/tsc_capture_control_ptx is not supported
[   0.1679 ] ERROR: /misc/tsc_controls/tsc_capture_config_ptx is not supported
[   0.1679 ] ERROR: /misc/tsc_controls/tsc_stscrsr is not supported
[   0.1679 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_num_control is not supported
[   0.1679 ] 
[   0.1679 ] Parsing config file :tegra234-mb1-bct-device-p3701-0000_cpp.dtb 
[   0.1679 ] Added Platform Config 9 data with size :- 100
[   0.1679 ] 
[   0.1682 ] Updating mb1-bct with firmware information
[   0.1717 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct_MB1.bct --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.1755 ] tegrahost_v2 --chip 0x23 0 --align mb1_cold_boot_bct_MB1_aligned.bct
[   0.1775 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_cold_boot_bct_MB1_aligned.bct zerosbk
[   0.1783 ] adding BCH for mb1_cold_boot_bct_MB1_aligned.bct
[   0.1808 ] tegrasign_v3.py --key None --list mb1_cold_boot_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1810 ] Assuming zero filled SBK key
[   0.1833 ] Warning: pub_key.key is not found
[   0.1834 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_cold_boot_bct_MB1_aligned_sigheader.bct.encrypt mb1_cold_boot_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.1852 ] Generating recovery mb1-bct
[   0.1862 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct.cfg --misc tegra234-mb1-bct-misc-p3701-0000_cpp.dtb --device tegra234-mb1-bct-device-p3701-0000_cpp.dtb
[   0.1871 ] MB1-BCT version: 0.10
[   0.1872 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED1/ is not supported
[   0.1878 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED1/ is not supported
[   0.1883 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED1/ is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_locking_config is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_locking_diff_configuration is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_locking_ref_frequency_configuration is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_locking_control is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_configuration is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_locking_fast_adjust_configuration is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_delta_control is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_capture_control_ptx is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_capture_config_ptx is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_stscrsr is not supported
[   0.1886 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_num_control is not supported
[   0.1886 ] 
[   0.1886 ] Parsing config file :tegra234-mb1-bct-device-p3701-0000_cpp.dtb 
[   0.1886 ] Added Platform Config 9 data with size :- 100
[   0.1886 ] 
[   0.1887 ] Updating mb1-bct with firmware information
[   0.1898 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct_MB1.bct --recov --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.1925 ] tegrahost_v2 --chip 0x23 0 --align mb1_bct_MB1_aligned.bct
[   0.1943 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_bct_MB1_aligned.bct zerosbk
[   0.1952 ] adding BCH for mb1_bct_MB1_aligned.bct
[   0.1978 ] tegrasign_v3.py --key None --list mb1_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1979 ] Assuming zero filled SBK key
[   0.1998 ] Warning: pub_key.key is not found
[   0.1992 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_bct_MB1_aligned_sigheader.bct.encrypt mb1_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.2037 ] Error: Skip generating mem_bct because sdram_config is not defined
[   0.2037 ] Error: Skip generating mem_bct because sdram_config is not defined
[   0.2037 ] Copying signatures
[   0.2085 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.2191 ] mb1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.2193 ] psc_bl1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.2193 ] Boot Rom communication
[   0.2209 ] tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[   0.2219 ] BR_CID: 0x80012344705DD8108400000009FE0140
[   0.3298 ] Sending bct_br
[   0.4318 ] ERROR: might be timeout in USB write.
Error: Return value 3
Command tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
Reading board information failed.

To troubleshoot, here have been my steps:

  1. Replaced the USB-C cable between the orin and host - no change
  2. Reflashed another Orin dev kit using the same image without the OTA changes and was successful
  3. Attempted to flash other jetson versions and images - no change

I believe I have narrowed the issue down to the OTA changes I made. I have never come across this error before when flashing.

My question is, why would this behaviour occur and how do I fix it?

3

Hi,

what do you mean with OTA changes I made?
You mean some OTA tools that were not included in the default BSP?
I don’t think that makes any difference.

Can you please check the serial console is not outputting anything when this error happens?
If that’s the case, then this a bug that has been around for a while.
If you are using a Ubuntu 20.04 PC, please try again with 18.04, try with different USB cables/ports, or try this command:

sudo -s
echo -1 > /sys/module/usbcore/parameters/autosuspend

4

I updated from Jetson 35.3.1 on the target board to Jetson 35.4.1 using an OTA update.

I tried flashing on another target board. The only difference between the two target boards I have tried is that one had the OTA update, while the other did not. The one without an OTA update was able to be successfully flashed using the same configuration and PC.

There is no serial console output during flashing.

There is no change and the flashing still fails with the same error.

5

You mean Debian-based OTA or Image-based OTA?
But either way, it should not be making any difference, and if the timeout error keeps happening,
this is all we could suggest currently:

Based on the observation, it is what many user have faced, and we are still debugging on it.

6

Image-based OTA

I tried swapping the USB, using ubuntu 18.04, and all other suggestions for these related posts when it shows a timeout error is occuring. None of these fixes it, and the same error occurs.

I have narrowed down the issue to the OTA update, and have had no issues flashing before this. Is it possible that the OTA update could be affecting recovery mode in some way?

7

I just don’t think it is related to OTA.
Maybe leave the device for a couple days, and flashing works after you get back to it.

8

I do not see how this could possibly be the solution.

In that case would you be able to verify this by conducting an ota from 35.3.1 to 35.4.1 on your own device and see if you are able to flash afterwards?

9

That’s what has happened on our side before…
We wanted to debug with capturing the USB signal, but when we got the equipment a few days later, the issue was gone.

I’ve said, this issue can happen all of a sudden without any sign, and we have tons of forum topics asking about this issue, which is mostly not about OTA.

Can you try if changing a host PC works?

10

Hi @wrogers

I notice it could be host side USB controller runs the auto-suspend function and cause the recovery mode has some problem.

This is kind of random behavior so you might think it is related to OTA, but that may not be related to it.
I also saw other kinds of random behavior before. Digging into OTA shall be a wrong direction.

Please try to disable the auto-suspend thing on your host PC, after that. Remove your usb cable, put the jetson into recovery mode again (must) and connect the cable back.

11

I was able to get the board to reflash properly by swapping the active bootloader using nvbootctrl: Update and Redundancy — Jetson Linux Developer Guide documentation

After OTA, bootloader B was set to active, running 35.4.1.

When swapping to the default bootloader A, it shows 35.3.1.

After swapping to bootloader A I was then able to reflash the board.

I think this is definitely an issue with OTA.