Flashing Jetson AGX Orin devkit after breaking system

Hello,

While I was using the Jetson Orin with the out of the box set up I executed by mistake a “sudo rm -r *” command. I stopped right after entering it but the damage was already done.
After that point, the system stopped working and not even a restart would solve it. This brings me to the conclusion that some relevant system files must have been erased.
When restarting the Jetson Orin I get the message in the booting screen as shown in the following image.

Then, I tried to flash the the OS again using the SDK Manager but I am getting the same issues as other people has been facing in older posts. I tried running the command ./flash.sh but also failed. My issue is that there is no way I can access the Jetson Orin.

###############################################################################
# L4T BSP Information:
# R35 , REVISION: 1.0
###############################################################################
# Target Board Information:
# Name: jetson-agx-orin-devkit, Board Family: t186ref, SoC: Tegra 234, 
# OpMode: production, Boot Authentication: NS, 
# Disk encryption: disabled ,
###############################################################################
copying emc_fuse_dev_params(/home/jgc/nvidia/nvidia_sdk/JetPack_5.0.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-br-bct-diag-boot.dts)... done.
copying device_config(/home/jgc/nvidia/nvidia_sdk/JetPack_5.0.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-mb1-bct-device-p3701-0000.dts)... done.
copying misc_config(/home/jgc/nvidia/nvidia_sdk/JetPack_5.0.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/t186ref/BCT/tegra234-mb1-bct-misc-p3701-0000.dts)... done.
./tegraflash.py --chip 0x23 --applet "/home/jgc/nvidia/nvidia_sdk/JetPack_5.0.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/mb1_t234_prod.bin" --skipuid --cfg readinfo_t234_min_prod.xml --dev_params tegra234-br-bct-diag-boot.dts --device_config tegra234-mb1-bct-device-p3701-0000.dts --misc_config tegra234-mb1-bct-misc-p3701-0000.dts --bins "mb2_applet applet_t234.bin" --cmd "dump eeprom cvm cvm.bin; dump custinfo custinfo_out.bin; reboot recovery" 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0074 ] tegrarcm_v2 --chip 0x23 0 --ismb2applet
[   0.0080 ] File rcm_state open failed
[   0.0083 ] ERROR: failed to read rcm_state
[   0.0083 ] 
[   0.0091 ] tegrasign_v3.py --key None --getmode mode.txt
[   0.0093 ] Assuming zero filled SBK key
[   0.0086 ] Pre-processing config: tegra234-mb1-bct-device-p3701-0000.dts
[   0.0135 ] Pre-processing config: tegra234-mb1-bct-misc-p3701-0000.dts
[   0.0202 ] Parsing partition layout
[   0.0210 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.0223 ] Parsing partition layout
[   0.0230 ] tegraparser_v2 --pt readinfo_t234_min_prod.xml.tmp
[   0.0238 ] Creating list of images to be signed
[   0.0245 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --list images_list.xml zerosbk
[   0.0251 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.0258 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.0261 ] MB1: Nvheader already present is mb1_t234_prod_aligned.bin
[   0.0302 ] Header already present for mb1_t234_prod_aligned_sigheader.bin
[   0.0305 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.0343 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.0346 ] adding BCH for mb2_t234_aligned.bin
[   0.0370 ] MB1: Nvheader already present is psc_bl1_t234_prod_aligned.bin
[   0.0490 ] Header already present for psc_bl1_t234_prod_aligned_sigheader.bin
[   0.0493 ] adding BCH for mb2_t234_aligned.bin
[   0.0633 ] Generating signatures
[   0.0641 ] tegrasign_v3.py --key None --list images_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.0643 ] Assuming zero filled SBK key
[   0.0737 ] Warning: pub_key.key is not found
[   0.0731 ] Parsing dev params for multi chains
[   0.0781 ] Generating br-bct
[   0.0788 ] Updating dev and MSS params in BR BCT
[   0.0788 ] tegrabct_v2 --dev_param tegra234-br-bct-diag-boot_cpp.dtb --brbct br_bct.cfg --chip 0x23 0
[   0.0797 ] Updating bl info
[   0.0804 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updateblinfo readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.0810 ] WARNING: boot chain is not completed. set to 0
[   0.0816 ] Get Signed section of bct
[   0.0823 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --listbct bct_list.xml
[   0.0830 ] Signing BCT
[   0.0838 ] tegrasign_v3.py --key None --list bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.0840 ] Assuming zero filled SBK key
[   0.0861 ] Sha saved in br_bct_BR.sha
[   0.0864 ] Warning: pub_key.key is not found
[   0.0857 ] Updating BCT with signature
[   0.0863 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesig bct_list_signed.xml
[   0.0869 ] Offset :4608 Len :3584
[   0.0872 ] Generating SHA2 Hash
[   0.0880 ] tegrasign_v3.py --key None --list bct_list.xml --sha sha512
[   0.0881 ] Assuming zero filled SBK key
[   0.0882 ] Assuming zero filled SBK key
[   0.0902 ] Sha saved in br_bct_BR.sha
[   0.0897 ] Updating BCT with SHA2 Hash
[   0.0904 ] tegrabct_v2 --brbct br_bct_BR.bct --chip 0x23 0 --updatesha bct_list_signed.xml
[   0.0910 ] Offset :4608 Len :3584
[   0.0912 ] Offset :68 Len :8124
[   0.0914 ] Generating coldboot mb1-bct
[   0.0921 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct.cfg --misc tegra234-mb1-bct-misc-p3701-0000_cpp.dtb --device tegra234-mb1-bct-device-p3701-0000_cpp.dtb
[   0.0927 ] MB1-BCT version: 0.9

[   0.0938 ] Parsing config file :tegra234-mb1-bct-device-p3701-0000_cpp.dtb 
[   0.0940 ] Added Platform Config 9 data with size :- 80
[   0.0940 ] 
[   0.0940 ] Updating mb1-bct with firmware information
[   0.0946 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct_MB1.bct --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.0963 ] tegrahost_v2 --chip 0x23 0 --align mb1_cold_boot_bct_MB1_aligned.bct
[   0.0978 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_cold_boot_bct_MB1_aligned.bct zerosbk
[   0.0985 ] adding BCH for mb1_cold_boot_bct_MB1_aligned.bct
[   0.1000 ] tegrasign_v3.py --key None --list mb1_cold_boot_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1002 ] Assuming zero filled SBK key
[   0.1016 ] Warning: pub_key.key is not found
[   0.1016 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_cold_boot_bct_MB1_aligned_sigheader.bct.encrypt mb1_cold_boot_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.1026 ] Generating recovery mb1-bct
[   0.1032 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct.cfg --misc tegra234-mb1-bct-misc-p3701-0000_cpp.dtb --device tegra234-mb1-bct-device-p3701-0000_cpp.dtb
[   0.1039 ] MB1-BCT version: 0.9

[   0.1049 ] Parsing config file :tegra234-mb1-bct-device-p3701-0000_cpp.dtb 
[   0.1051 ] Added Platform Config 9 data with size :- 80
[   0.1051 ] 
[   0.1051 ] Updating mb1-bct with firmware information
[   0.1058 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_bct_MB1.bct --recov --updatefwinfo readinfo_t234_min_prod.xml.bin
[   0.1073 ] tegrahost_v2 --chip 0x23 0 --align mb1_bct_MB1_aligned.bct
[   0.1087 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --appendsigheader mb1_bct_MB1_aligned.bct zerosbk
[   0.1093 ] adding BCH for mb1_bct_MB1_aligned.bct
[   0.1107 ] tegrasign_v3.py --key None --list mb1_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[   0.1108 ] Assuming zero filled SBK key
[   0.1121 ] Warning: pub_key.key is not found
[   0.1121 ] tegrahost_v2 --chip 0x23 0 --updatesigheader mb1_bct_MB1_aligned_sigheader.bct.encrypt mb1_bct_MB1_aligned_sigheader.bct.hash zerosbk
[   0.1132 ] Error: Skip generating mem_bct because sdram_config is not defined
[   0.1132 ] Error: Skip generating mem_bct because sdram_config is not defined
[   0.1132 ] Copying signatures
[   0.1139 ] tegrahost_v2 --chip 0x23 0 --partitionlayout readinfo_t234_min_prod.xml.bin --updatesig images_list_signed.xml
[   0.1179 ] mb1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.1181 ] psc_bl1_t234_prod_aligned_sigheader.bin.encrypt filename is from images_list
[   0.1181 ] Boot Rom communication
[   0.1188 ] tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[   0.1195 ] BR_CID: 0x80012344705DD5176C00000006FE0080
[   0.2737 ] Sending bct_br
[   0.4198 ] ERROR: might be timeout in USB write.
Error: Return value 3
Command tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
Reading board information failed.

I also tried from the SDK and this is the following error I get when I use the Automatic Setup.

I then tried running it with Manual setup and I also got an error:
SDKM_logs_JetPack_5.0.2_(rev._1)_Linux_for_Jetson_AGX_Orin_modules_2023-01-09_16-55-51.zip (332.6 KB)

I all cases I was running the Jetson Orin in Recovery Mode pressing the reset button and while that pressing the restart and releasing both at the same time.

I don’t know what else can I try to flash it or to have the Orin working again.
I hope I have provided enough information to solve this issue.

Thank you very much for your time.
Jordi

Hi jordigc2,

It looks like some system files has been removed with sudo rm -r * command.

Are you using the standalone Ubuntu 18.04 as your host PC for SDKM and flash script?

You should release restart button before recovery button to enter force recovery mode.
You could run lsusb on your host PC to make sure if your board is in force recovery mode.
To Determine Whether the Developer Kit Is in Force Recovery Mode

I have similar situation as your, change port, change cable still “ERROR: might be timeout in USB write.”
I tried this and can flash, hope it may help you :

  • Hold Recovery button, plug in your TypeC cable to an USB2 port
  • Plugin power, it may boot your Orin, if not press Power to boot it. Check lsusb if see 7023 device
  • Here is the step i done after the above two steps still failed . Hold the Recovery button and press Reset, keep holding Recovery button for maybe 3s. Then try again, this help me flash my Orin AGX devkit
1 Like

Hello,

I have set it properly to the recovery mode as you can see in the following image:

I am using Ubuntu 20.04 PC for flashing the Orin.
I have tried flashing it again now that I new that it was in recovery mode and the same errors as mentioned before happened.
Here I attach the new log file from when I was doing the Manual SetUp:
SDKM_logs_JetPack_5.0.2_(rev._1)_Linux_for_Jetson_AGX_Orin_modules_2023-01-10_09-08-51.zip (77.2 KB)

Thanks for your time

I found there are some topics on the forum about this issue.
Most of them could resolve this issue with Ubuntu 18.04 host PC.

It looks like kernel version of host PC causing this issue.
USB Timeout when flashing AGX Orin - #14 by guotl321
Could you help to provide the result of uname -r on your host PC.

and Can you run apt-get update on your host, reboot and test again?

The output from the uname -r is:

5.14.0-1055-oem

I have also updated and rebooted the host machine and still facing the exact same issues.

Yes, it looks like that you are using the same kernel version (5.14.0) as that post.

There are two ways could resolve this issue at this moment, one is using Ubuntu18.04, another is updating kernel to 5.15.0 for Ubuntu 20.04.

I have updated the kernel, but using the Ubuntu Mainline application I could not find the specific version mentioned in that post but this one:

Linux 002036-ws 5.15.0-57-generic #63~20.04.1-Ubuntu SMP Wed Nov 30 13:40:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

And I have tested it again and it still fails.

I have also tried with kernel version :

Linux 002036-ws 5.15.0-051500-generic #202110312130 SMP Sun Oct 31 21:33:20 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

And got the same issue.

Please find attached the new logs:
SDKM_logs_JetPack_5.0.2_(rev._1)_Linux_for_Jetson_AGX_Orin_modules_2023-01-10_10-54-52.zip (137.7 KB)

From flash log of SDKM, it looks the same as before.
Please also provide the serial console log of the board when flashing for further check.

Could you also provide the result of lsb_release -a from your host PC?

Do you have standalone or dual boot Ubuntu 18.04 host PC to flash the board?

1 Like

The output of the lsb_release -a is the following:

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal

Unfortunately I do not have an Ubuntu 18.04 available with me.

There are many users resolving this issue with Ubuntu18.04. If you could access any Ubuntu 18.04 host PC, please give it a try!

I’ve investigated in flash process, this error is directly coming from USB read from BootRom.

Could you help to try using other USB cable?
Other USB port or reboot your host PC.

And we want to check the serial console log while flashing the board.

This is the output of dmesg when connecting the USB connected to the Jetson Orin in recovery mode and flashing the JetPack:

[104449.700096] usb 1-1: new high-speed USB device number 56 using xhci_hcd
[104449.849447] usb 1-1: New USB device found, idVendor=0955, idProduct=7023, bcdDevice= 4.01
[104449.849463] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[104449.849469] usb 1-1: Product: APX
[104449.849473] usb 1-1: Manufacturer: NVIDIA Corp.
[106106.123601] bpfilter: Loaded bpfilter_umh pid 135899
[106106.123749] bpfilter: read fail 0
[106106.254328] bpfilter: Loaded bpfilter_umh pid 135901
[106106.254795] bpfilter: read fail 0
[106106.354351] bpfilter: Loaded bpfilter_umh pid 135903
[106106.354807] bpfilter: read fail 0
[106106.450366] bpfilter: Loaded bpfilter_umh pid 135905
[106106.450815] bpfilter: read fail 0
[106106.534628] bpfilter: Loaded bpfilter_umh pid 135908
[106106.535087] bpfilter: read fail 0
[106106.631661] bpfilter: Loaded bpfilter_umh pid 135912
[106106.632082] bpfilter: read fail 0
[106106.721971] bpfilter: Loaded bpfilter_umh pid 135914
[106106.722438] bpfilter: read fail 0
[106106.830180] bpfilter: Loaded bpfilter_umh pid 135916
[106106.830645] bpfilter: read fail 0
[106106.908687] bpfilter: Loaded bpfilter_umh pid 135918
[106106.908851] bpfilter: read fail 0
[106107.024570] bpfilter: Loaded bpfilter_umh pid 135922
[106107.024719] bpfilter: read fail 0
[106107.131358] bpfilter: Loaded bpfilter_umh pid 135924
[106107.131760] bpfilter: read fail 0
[106107.229853] bpfilter: Loaded bpfilter_umh pid 135926
[106107.230326] bpfilter: read fail 0
[106107.298041] bpfilter: Loaded bpfilter_umh pid 135928
[106107.298504] bpfilter: read fail 0
[106107.402182] bpfilter: Loaded bpfilter_umh pid 135930
[106107.402611] bpfilter: read fail 0
[106107.514756] bpfilter: Loaded bpfilter_umh pid 135932
[106107.515213] bpfilter: read fail 0
[106107.642143] bpfilter: Loaded bpfilter_umh pid 135935
[106107.642599] bpfilter: read fail 0
[106107.730400] bpfilter: Loaded bpfilter_umh pid 135937
[106107.730861] bpfilter: read fail 0
[106107.814195] bpfilter: Loaded bpfilter_umh pid 135939
[106107.814658] bpfilter: read fail 0
[106107.914206] bpfilter: Loaded bpfilter_umh pid 135941
[106107.914656] bpfilter: read fail 0
[106108.018491] bpfilter: Loaded bpfilter_umh pid 135943
[106108.018939] bpfilter: read fail 0
[106109.997986] xhci_hcd 0000:3a:00.0: xHC error in resume, USBSTS 0x401, Reinit
[106109.998000] usb usb3: root hub lost power or was reset
[106109.998006] usb usb4: root hub lost power or was reset

The exact same error happens with a different USB cable.

It looks like USB device could be recognized at this moment.

How could you get this dmesg?
Aren’t you still unable to flash the board? (with ERROR: might be timeout in USB write)

Do you still have flash failed issue?

I get that dmesg when I connect the board in recovery mode. And as soon as I start flashing the JetPack version I get the subsequent error messages.

As I am not able to flash it using a 20.04 I am trying to get a host with ubuntu 18.04.

It’s weird that it’s not possible to flash the jetson orin from ubuntu 20.04… Do you know which could be the reason?

Could you provide the full flash log and serial console log at the same time for further check?

We found some users getting this issue (timeout in USB write) with Ubuntu 20.04. Some users could resolve with newer kernel version (5.15.0 as you updated), but some users not.
This release (Ubuntu 20.04) should be available for flashing the board with Jetpack 5.0.2.

We could not reproduce this issue at our side and still figure out if it might relate to the specific USB driver of host PC.
Do you have other host PC (different brand/vendor) with Ubuntu 20.04?

I have shared all the possible logs that I could find.

Inside the company we all have DELL computers, but in the next days I should get a new PC from a different brand which I will try. It will come with Ubuntu20.04 as well but hopefully it will not be a problem. If not Next step will be to build a PC with Ubuntu18.04.

Thanks for your time

Thanks for your trials on this issue. Look forward your result.

Hello Kevin,

I managed to get a spare PC and install Ubuntu 18.04 in it.

After flashing the JetPack on the orin I could see that some packages were skipped as you can see in he attached image. Is this normal or they should have been installed?

An this window popped up right after:

After flash the Jetson reboots. Once first boot account setup is complete those extra packages are flashed via ssh over the network. The message you are seeing suggests the network is not configured correctly and networking does not succeed. I don’t know if this is a misleading message or not, but network failure would account for the packages not reaching the Jetson.

That default address of 192.168.55.1 is via a virtual ethernet device. The recovery mode Jetson does not have this, but after flash and reboot, the USB cable provides a virtual network device with a DHCP server. That device creates address 192.168.55.100 on the host PC, and address 192.168.55.1 on the Jetson (thus the default in the GUI). In theory the host PC could:

  • ping 192.168.55.1 (to ping the Jetson)
  • ping 192.168.55.100 (to ping itself)

And the Jetson could:

  • ping 192.168.55.1 (to ping itself)
  • ping 192.168.55.100 (to ping the host PC)

Ping could be replaced by ssh. However, this only works if:

  • The Jetson rebooted to provide the virtual device.
  • The Jetson’s first boot account setup completed (to give an admin login).
  • The PC accepts this device, allowing configuration.
  • The PC is not firewalled against this.

Most of the time those requirements work without issue. I’ve not seen the error message you mention though. It makes me think there might be more than one device at address 192.168.55.1. If the Jetson has wired ethernet attached to the same switch/router that the host PC has, then you could use that address instead of 192.168.55.1 (and it’d even be faster if it is gigabit).

Do note that you can run sdkmanager again, deselect flash, and deselect adding host PC components. And then just attempt install of the other components without flashing. Can you log in to the Jetson, and see what it says from:

  • ifconfig (or ip -s a)
  • route (or ip route)