Flash failed. Orin no more recoverable or flashable

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hello

During flashing 6.0.8.1 via docker the flashing ended with flashing failed. The orin is no more booting and its in blank screen I tried sdk manager to bring it to 6.0.6 but flashing is failing here too. I have attached logs here. I am able to ssh to aurix via ACM1 . But I can’t connect to ACM2 and 3. So essentially i believe SOC is down. Is there any way to recover the system.?

[2024-02-09 11:19:35,185 root DEBUG console_logger.py 17 30611] b'\r\n[bootburnTegra-A]: [AdbPush(327)] : [AdbPush] failed retrying...'

[2024-02-09 11:19:35,187 root DEBUG console_logger.py 17 30611] b'\r\n'

[2024-02-09 11:19:35,191 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(163)] : shell command -- /drive/drive-foundation/tools/flashtools/flash/adb -s 8S2TQ6G1G00000401000 shell /tmp/wr_sh.sh rm /tmp/A_6_mb2_t234_zerosign.bin failed'

[2024-02-09 11:19:35,192 root DEBUG console_logger.py 17 30611] b'\r\n'

[2024-02-09 11:19:35,193 root DEBUG console_logger.py 17 30611] b"[bootburnTegra-A]: [executeShellCommand(165)] : Output: error: device '8S2TQ6G1G00000401000' not found"

[2024-02-09 11:19:35,194 root DEBUG console_logger.py 17 30611] b'\r\n\r\n'

[2024-02-09 11:19:35,195 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(172)] : Return code: 1'

[2024-02-09 11:19:35,197 root DEBUG console_logger.py 17 30611] b'\r\n'

[2024-02-09 11:19:35,200 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(163)] : shell command -- /drive/drive-foundation/tools/flashtools/flash/adb -s 8S2TQ6G1G00000401000 push /drive/drive-foundation/tools/flashtools/bootburn_t23x_py/__temp_t234/Tegra-A-0dc8573a7bf/flash-images/2500000.ufshci:0_38 /tmp failed\r\n'

[2024-02-09 11:19:35,201 root DEBUG console_logger.py 17 30611] b"[bootburnTegra-A]: [executeShellCommand(165)] : Output: adb: error: failed to get feature set: device '8S2TQ6G1G00000401000' not found\r\n\r\n"

[2024-02-09 11:19:35,202 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(172)] : Return code: 1\r\n'

[2024-02-09 11:19:35,204 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [AdbPush(327)] : [AdbPush] failed retrying...'

[2024-02-09 11:19:35,206 root DEBUG console_logger.py 17 30611] b'\r\n[bootburnTegra-A]: [executeShellCommand(163)] : shell command -- /drive/drive-foundation/tools/flashtools/flash/adb -s 8S2TQ6G1G00000401000 push /drive/drive-foundation/tools/flashtools/bootburn_t23x_py/__temp_t234/Tegra-A-0dc8573a7bf/flash-images/3460000.sdhci_40 /tmp failed\r\n'

[2024-02-09 11:19:35,206 root DEBUG console_logger.py 17 30611] b"[bootburnTegra-A]: [executeShellCommand(165)] : Output: adb: error: failed to get feature set: device '8S2TQ6G1G00000401000' not found"

[2024-02-09 11:19:35,207 root DEBUG console_logger.py 17 30611] b'\r\n\r\n[bootburnTegra-A]: [executeShellCommand(172)] : Return code: 1'

[2024-02-09 11:19:35,208 root DEBUG console_logger.py 17 30611] b'\r\n[bootburnTegra-A]: [AdbPush(327)] : [AdbPush] failed retrying...\r\n'

[2024-02-09 11:19:35,211 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(163)] : shell command -- /drive/drive-foundation/tools/flashtools/flash/adb -s 8S2TQ6G1G00000401000 push /drive/drive-foundation/tools/flashtools/bootburn_t23x_py/__temp_t234/Tegra-A-0dc8573a7bf/flash-images/2500000.ufshci:0_38 /tmp failed\r\n'

[2024-02-09 11:19:35,212 root DEBUG console_logger.py 17 30611] b"[bootburnTegra-A]: [executeShellCommand(165)] : Output: adb: error: failed to get feature set: device '8S2TQ6G1G00000401000' not found\r\n\r\n"

[2024-02-09 11:19:35,212 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(172)] : Return code: 1\r\n'

[2024-02-09 11:19:35,213 root DEBUG console_logger.py 17 30611] b"command line used was:\r\n['/drive/drive-foundation//tools/flashtools/bootburn/bootburn.py', '-b', 'p3710-10-s05', '-B', 'qspi', '-x', '/dev/ttyACM1']\r\n\r\n\r\n\x1b[01;31mAdb push failed -- /drive/drive-foundation/tools/flashtools/flash/adb -s 8S2TQ6G1G00000401000 push /drive/drive-foundation/tools/flashtools/bootburn_t23x_py/__temp_t234/Tegra-A-0dc8573a7bf/flash-images/2500000.ufshci:0_38 /tmp\x1b[0m\r\n\r\n"

[2024-02-09 11:19:35,214 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(163)] : shell command -- /drive/drive-foundation/tools/flashtools/flash/adb -s 8S2TQ6G1G00000401000 push /drive/drive-foundation/tools/flashtools/bootburn_t23x_py/__temp_t234/Tegra-A-0dc8573a7bf/flash-images/3460000.sdhci_40 /tmp failed\r\n'

[2024-02-09 11:19:35,214 root DEBUG console_logger.py 17 30611] b"[bootburnTegra-A]: [executeShellCommand(165)] : Output: adb: error: failed to get feature set: device '8S2TQ6G1G00000401000' not found\r\n\r\n"

[2024-02-09 11:19:35,215 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(172)] : Return code: 1\r\n'

[2024-02-09 11:19:35,215 root DEBUG console_logger.py 17 30611] b"command line used was:\r\n['/drive/drive-foundation//tools/flashtools/bootburn/bootburn.py', '-b', 'p3710-10-s05', '-B', 'qspi', '-x', '/dev/ttyACM1']\r\n\r\n\r\n\x1b[01;31mAdb push failed -- /drive/drive-foundation/tools/flashtools/flash/adb -s 8S2TQ6G1G00000401000 push /drive/drive-foundation/tools/flashtools/bootburn_t23x_py/__temp_t234/Tegra-A-0dc8573a7bf/flash-images/3460000.sdhci_40 /tmp\x1b[0m\r\n\r\n"

[2024-02-09 11:19:35,223 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(163)] : shell command -- /drive/drive-foundation/tools/flashtools/flash/adb -s 8S2TQ6G1G00000401000 shell /tmp/wr_sh.sh /tmp/nvdd --device /dev/block/3270000.spi --startoffset 49283072 --partsize 437264 --md5sum 845391d3fa4d5c4eda564674af8d135f --printmd5sum failed\r\n'

[2024-02-09 11:19:35,224 root DEBUG console_logger.py 17 30611] b"[bootburnTegra-A]: [executeShellCommand(165)] : Output: error: device '8S2TQ6G1G00000401000' not found\r\n\r\n"

[2024-02-09 11:19:35,225 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [executeShellCommand(172)] : Return code: 1\r\n'

[2024-02-09 11:19:35,225 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [checkMd5(442)] : MD5 for partition A_6_mb2_t234_zerosign.bin does not match ... written image corrupted\r\n'

[2024-02-09 11:19:35,226 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [checkMd5(443)] : expected MD5 sum = 845391d3fa4d5c4eda564674af8d135f\r\n'

[2024-02-09 11:19:35,227 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [checkMd5(444)] : \r\n\x1b[01;31m117\r\n\x1b[0m'

[2024-02-09 11:19:35,227 root DEBUG console_logger.py 17 30611] b'\r\n'

[2024-02-09 11:19:35,228 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [SendFileUsingADB(631)] : Target image written incorrectly ... corrupted\r\n'

[2024-02-09 11:19:35,228 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [FlashUsingADB(1005)] : Failed to correctly write partition A_mb2-bootloader\r\n'

[2024-02-09 11:19:35,229 root DEBUG console_logger.py 17 30611] b"command line used was:\r\n['/drive/drive-foundation//tools/flashtools/bootburn/bootburn.py', '-b', 'p3710-10-s05', '-B', 'qspi', '-x', '/dev/ttyACM1']\r\n\r\n\r\n\x1b[01;31mBad MD5 Sum on A_mb2-bootloader\x1b[0m\r\n\r\n"

[2024-02-09 11:19:35,229 root DEBUG console_logger.py 17 30611] b'Process Process-1:1:\r\n'

[2024-02-09 11:19:35,230 root DEBUG console_logger.py 17 30611] b'Traceback (most recent call last):\r\n'

[2024-02-09 11:19:35,230 root DEBUG console_logger.py 17 30611] b' File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap\r\n self.run()\r\n File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run\r\n self._target(*self._args, **self._kwargs)\r\n File "/drive/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_adb.py", line 1006, in FlashUsingADB\r\n AbnormalTermination("Bad MD5 Sum on " + partitionInfo.PartitionName, nverror.NvError_Bad_MD5)\r\n File "/drive/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/flashtools_nverror.py", line 251, in AbnormalTermination\r\n raise OSError(errorCode)\r\nOSError: 55\r\n'

SDKM_logs_DRIVE_OS_6.0.6_SDK_Linux_for_DRIVE_AGX_Orin_DevKits_2024-02-09_16-01-31.zip (545.3 KB)

log.txt (1.5 MB)

Dear @jishnuw,

For Tegra, You need to connect to /dev/ttyACM0.

What is the DRIVE OS version on board currently? Was it in working condition before flash.

Hi @SivaRamaKrishnaNV

Even ACM0 is not connecting
image

Yes before flashing it was working pretty fine. It was on 6.0.8.1. We wanted to re flash.

[2024-02-09 11:19:35,225 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [checkMd5(442)] : MD5 for partition A_6_mb2_t234_zerosign.bin does not match ... written image corrupted\r\n'
[2024-02-09 11:19:35,226 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [checkMd5(443)] : expected MD5 sum = 845391d3fa4d5c4eda564674af8d135f\r\n'
[2024-02-09 11:19:35,227 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [checkMd5(444)] : \r\n\x1b[01;31m117\r\n\x1b[0m'
[2024-02-09 11:19:35,227 root DEBUG console_logger.py 17 30611] b'\r\n'
[2024-02-09 11:19:35,228 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [SendFileUsingADB(631)] : Target image written incorrectly ... corrupted\r\n'
[2024-02-09 11:19:35,228 root DEBUG console_logger.py 17 30611] b'[bootburnTegra-A]: [FlashUsingADB(1005)] : Failed to correctly write partition A_mb2-bootloader\r\n'
[2024-02-09 11:19:35,229 root DEBUG console_logger.py 17 30611] b"command line used was:\r\n['/drive/drive-foundation//tools/flashtools/bootburn/bootburn.py', '-b', 'p3710-10-s05', '-B', 'qspi', '-x', '/dev/ttyACM1']\r\n\r\n\r\n\x1b[01;31mBad MD5 Sum on A_mb2-bootloader\x1b[0m\r\n\r\n"
[2024-02-09 11:19:35,229 root DEBUG console_logger.py 17 30611] b'Process Process-1:1:\r\n'
[2024-02-09 11:19:35,230 root DEBUG console_logger.py 17 30611] b'Traceback (most recent call last):\r\n'
[2024-02-09 11:19:35,230 root DEBUG console_logger.py 17 30611] b'  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap\r\n    self.run()\r\n  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run\r\n    self._target(*self._args, **self._kwargs)\r\n  File "/drive/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_adb.py", line 1006, in FlashUsingADB\r\n    AbnormalTermination("Bad MD5 Sum on " + partitionInfo.PartitionName, nverror.NvError_Bad_MD5)\r\n  File "/drive/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/flashtools_nverror.py", line 251, in AbnormalTermination\r\n    raise OSError(errorCode)\r\nOSError: 55\r\n'
[2024-02-09 11:33:58,178 root DEBUG console_logger.py 17 30611] b'[bootburn]: \r\n\x1b[01;31mFlashing timed out -- Exiting\x1b[0m\r\n\r\n'
[2024-02-09 11:34:42,083 root DEBUG console_logger.py 17 30611] b'\r\n'
[2024-02-09 11:34:42,184 root ERROR pdk_flasher.py 280 30611] Flashing process exited with error 5
[2024-02-09 11:34:42,194 root INFO runner.py 37 30611] Error in flashing!
[2024-02-09 11:34:42,195 root DEBUG runner.py 39 30611] Error on line 990
[2024-02-09 11:34:42,195 root DEBUG runner.py 40 30611] Exception info: Exception Type <class 'module.errors.FlashingError'>, Traceback <traceback object at 0x7f87fb5e2fc0>
[2024-02-09 11:34:42,195 root INFO runner.py 42 30611] driveinstaller cannot continue! Check the log file for details: /drive/driveinstaller/driveinstaller.log
[2024-02-09 11:34:42,195 root DEBUG runner.py 62 30611] Cleaning up...
[2024-02-09 11:34:42,195 root DEBUG utilities.py 137 30611] Deleting temporary directories...
[2024-02-09 11:34:42,195 root DEBUG utilities.py 139 30611] Deleting /tmp/pdkinstall-tmp-o1fmkkkg-bind
[2024-02-09 11:34:42,196 root DEBUG runner.py 64 30611] Closing the log file.

Is there any way to recover the SOC? Any recovery mode? From the log I see the image is corrupted (from its md5 sum log). What can we do next?

Dear @jishnuw,

  1. Could you connect to aurix via minicom and run tegrareset x1 and see /dev/ttyACM0 is accessible.
  2. Connect host ↔ target connections(connect host to micro USB debug port, LEFT TYPE C USB port on target as mentioned in quick start guide) and share lsusb output on host.
  3. After #2, run tegrarecovery x1 on and tegrareset x1 on aurix console and share the lsusb output on host ( This sets the board in recovery mode and I would expect to see additional NVIDIA device from #2 output
  4. Keep the board back in normal by running tegrarecovery x1 off and tegrareset x1 on aurix console

Hi @SivaRamaKrishnaNV

  1. Could you connect to aurix via minicom and run tegrareset x1 and see /dev/ttyACM0 is accessible.

    Its not accessible. I see the dump.
  1. Connect host ↔ target connections(connect host to micro USB debug port, LEFT TYPE C USB port on target as mentioned in quick start guide) and share lsusb output on host.
    jishnu@SafeAi:~$ lsusb
    Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 003 Device 083: ID 0955:7045 NVIDIA Corp. Tegra On-Platform Operator
    Bus 003 Device 004: ID 13d3:5415 IMC Networks Integrated Camera
    Bus 003 Device 003: ID 06cb:00bd Synaptics, Inc.
    Bus 003 Device 006: ID 8087:0026 Intel Corp.
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  1. After #2, run tegrarecovery x1 on and tegrareset x1 on aurix console and share the lsusb output on host ( This sets the board in recovery mode and I would expect to see additional NVIDIA device from #2 output
    jishnu@SafeAi:~$ lsusb
    Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 003 Device 083: ID 0955:7045 NVIDIA Corp. Tegra On-Platform Operator
    Bus 003 Device 004: ID 13d3:5415 IMC Networks Integrated Camera
    Bus 003 Device 003: ID 06cb:00bd Synaptics, Inc.
    Bus 003 Device 006: ID 8087:0026 Intel Corp.
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  1. Keep the board back in normal by running tegrarecovery x1 off and tegrareset x1 on aurix console
    Done.

have tried sharing ACM0 log as well as much as i could capture.
I am not sure if we see more lsusb devices.

ACM0_log.txt (17.5 KB)

I

Dear @jishnuw,
Press ENTER on /dev/ttyACM0 console and see if you get login screen.

In step#3, you should an additional NVIDIA device like below when board is set to recovery mode. I see 0955:7023 device is missing.

Bus 002 Device 021: ID 0955:7045 NVIDIA Corp.
Bus 002 Device 022: ID 0955:7023 NVIDIA Corp.

Could you share the aurix log of tegrarecovery x1 on & tegrareset x1? Also please share host ↔ target connection pictures for confirmation.

Hi @SivaRamaKrishnaNV

I tried enter but i am pretty sure there was no keyboard input into ACM0 console.

There is a small log snippet in the image shared previously. The left side is Aurix console for step1.

I left office so can’t share you the image now but i can assure you I have a micro usb connected, type c connected on left port next to the eth port. I have also connected the ethernet port to my laptop. I will share a picture tomorrow.

Meanwhile if what I shared above is correct how do I proceed ahead. I did see some kernel panics in the log not sure if they are meant to be.

If I am unable to press enter how do we proceed?

How can I get the other usb device visible as ypu mentioned if my steps are correct?

I just want to be ready with a few steps so that I can start off tomorrow morning asap and then I will share all logs as possible.

Getting the orin back is on priority 1 for us so please do share as many steps as you can so that I can try my best to get it back.

Thanks for the understanding.

Jishnu

Dear @jishnuw,
I see unknown command on top of aurix log. Could you double check if there is any typo?

Hi @SivaRamaKrishnaNV

I think i ran tegrareset without x1 as args. first hence wrong cmd. I think line 5 is right command isn’t it?

We are looking at left side of the image right?

Hi @SivaRamaKrishnaNV

If you can share some points to check and try may be i have something to start with.

Thanks in advance!
Jishnu

Yes.
Please check below things.

  1. Please check setting the board in recovery and see if you notice additional NVIDIA device. Put the device back in normal mode.
  2. Change the host target connection wires and check changing the USB port on host side.
  3. Retry flashing the board flash.py script.
  4. Try manual flashing using bootburn
cd /drive/drive-foundation
make/bind_partitions -b p3710-10-s05 linux
#put the board in recovery mode(run tegrarecovery x1 on and tegrareset x1 on aurix)
tools/flashtools/bootburn/bootburn.py -b <board config> -B qspi
##put the board in normal mode(run tegrarecovery x1 off and tegrareset x1 on aurix)

Please power cycle both the host and target systems, then use the docker method to flash 6.0.8.1. If the problem persists, please provide the docker method logs. Thanks.

Dear @SivaRamaKrishnaNV and @VickNV

I can’t thank you enough for the support!! I am able to recover the device.

Here is what I changed.

  1. Even though what I used is a fresh usb to micro usb cable, which was flashing, it didn’t enumerate the device when put to recovery. So I got another new cable and that worked. So some micro usbs don’t work even though same helped me connect to Aurix and also helped me flash.
jishnu@SafeAi:~$ lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 005: ID 0955:7045 NVIDIA Corp. Tegra On-Platform Operator
Bus 003 Device 011: ID 0955:7023 NVIDIA Corp. APX
Bus 003 Device 003: ID 13d3:5415 IMC Networks Integrated Camera
Bus 003 Device 002: ID 06cb:00bd Synaptics, Inc. 
Bus 003 Device 006: ID 8087:0026 Intel Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  1. Once I was able to put it to recovery mode. I made a fresh clone of 6.0.8.1 and tried docker flashing. Before that I did ensure I hard rebooted both devices host and target like @VickNV mentioned. Then docker flashing worked. Also ensured minicom was closed properly else won’t support flashing.

Thanks again.
Jishnu

1 Like

Good to hear that you resolved the issue.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.