Flashing Drive AGX Orin fails

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.2.10884
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

I was able to successfully flash this Drive AGX Orin device several times before. However, after a power failure in our building it seems to have corrupted something on the Drive AGX Orin device. When I try flash it now, I get this error (I’ll post the full log in another message):

[‘tools/flashtools/bootburn/bootburn.py’, ‘-b’, ‘p3710-10-a01’, ‘-B’, ‘qspi’, ‘-U’, ‘/home/kenneth.chu/nvidia/drive-foundation/tools/flashtools/storage_configs/t23x/ufs-provision-p3710.cfg’]

Provision Failed

Exception 59 raised in bootburn_active
Traceback (most recent call last):
File “/home/kenneth.chu/nvidia/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/bootburn.py”, line 150, in bootburn_active
bootburnLib.UFSDeviceProvision(targetConfig.p_OutDirPath)
File “/home/kenneth.chu/nvidia/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/bootburn_lib.py”, line 3506, in UFSDeviceProvision
adbObj.ProvisionUFSDevice(p_TempDumpPath)
File “/home/kenneth.chu/nvidia/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/bootburn_adb.py”, line 1338, in ProvisionUFSDevice
AbnormalTermination(“Provision Failed”, nverror.NvError_FileWriteFailed)
File “/home/kenneth.chu/nvidia/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/flashtools_nverror.py”, line 249, in AbnormalTermination
raise OSError(errorCode)
OSError: 59
[bootburnTegra-A]: [updateFindBoardData(352)] : Connected Board Name details :: : chipSku is - ct00
returning to directory /home/kenneth.chu/nvidia/drive-foundation
Cleaning up …
Cleaning temp dir

2

log.txt (8.5 KB)

3

Dear @kenneth.chu,
However, after a power failure in our building it seems to have corrupted something on the Drive AGX Orin device

Does that mean, the board is not booting after power failure? Do you see if fans running? Are you able to access Aurix and tegra via serial console? Could you check flashing with sdkmanager as well?

4

The board is booting after power failure. I can access Aurix and issue the “tegrareset x1” command and it will reset the Orin SoC. The Orin SoC also starts, but get stuck in the boot process and does not respond. I originally saw this and that’s why I wanted to try to reflash the device.

I also tried using sdkmanager to flash the device and it also failed.

5

The Orin SoC also starts, but get stuck in the boot process and does not respond

Can you access Tegra A console via minicom? If so,. could you share the boot log(messages on Tegra A console via minicom when system reboots)?

Could you share the sdkm logs?

7

The sdkm log is attached.
sdkm-2023-04-21-12-30-52.log (3.6 MB)

8

Did you attempt to flash the DRIVE OS 6.0.6 Linux SDK instead of QNX? If so, could you please provide the full logs located in the ~/.nvsdkm directory?

9

Yes, the previously attached sdkm.log is from the attempt to flash Drive OS 6.0.6 Linux SDK.

10

Please share your ~/.nvsdkm directory including other relevant logs.

11

Here it is.
nvsdkm.tar.gz (10.7 MB)

12

We are checking internally and will get back to you.

13

I also get it
Target module : AGX-ORIN-64GB

Software Version : DRIVE OS 6.0.6

Target Operating System : Linux Jetpack 5.1.1

Hardware Platform : custom board

SDK Manager Version ： 1.9.2.10888

Host Machine Version ： native Ubuntu Linux 18.04 Host installed with SDK Manager？？？？

Flash result ： fail

Fail-log ：RCM state error

SDKM_logs_JetPack_5.1.1_(rev._1)_Linux_for_Jetson_AGX_Orin_modules_2023-04-24_18-31-33.zip (121.1 KB)

15

@gucj
Thanks for reaching out. Regarding your flash failure issue, it would be best to create a separate forum topic with your corporate email for further assistance. This will allow our team to provide you with more tailored support and address your issue more effectively.

Please let me know if you have any questions.

16

@kenneth.chu Can you please provide further information about the power failure in your building and what tasks the devkit is performing during the outage?

It seems that the logs do not match the process of flashing DRIVE OS 6.0.6 Linux SDK through SDK Manager. Can you please provide further details on the steps you took leading up to the issue of being unable to flash the unit?

17

I think I was running a test with Nsight Systems CLI to capture profiling data for our app at the time of power failure.

The log that you attached in the message above is from trying to flash QNX on the devkit after power failure. It failed, so then I tried to flash Linux on the devkit using sdkmanager. The logs from the nvsdkm.tar.gz attached above is from trying to flash Linux using sdkmanager.

18

The following snippet indicates that the UFS was not functioning properly. Can you confirm if the UFS has been provisioned? If so, would you be able to assist in running the command ‘./bootburn.py -b p3710-10-a01 -U /drive-foundation/tools/flashtools/storage_configs/t23x/ufs-provision-256gb.cfg -D’ on the host and provide the target side log via /dev/ttyACM0? Thank you.

[bootburnTegra-A]: [executeShellCommand(156)] : nvdd: /dev/block/2500000.ufshci:0 not found, looking by controller instance…\r\n:[E]: NvParseDeviceByControllerInstance : 191: Error DeviceNotFound (0x18)\r\n:[E]: NvddGetDeviceInfo : 28: Error DeviceNotFound (0x18)\r\n:[E]: nvdd : 460: Error DeviceNotFound (0x18)\r\n\r\nEXITCODE=24\r\n\r\n[bootburnTegra-A]: [AdbShell(280)] : EXITCODE = 24\r\ncommand line used was:\r\n[‘/home/rnd/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation//tools/flashtools/bootburn/bootburn.py’, ‘-b’, ‘p3710-10-s05’, ‘-B’, ‘qspi’, ‘-x’, ‘/dev/ttyACM1’, ‘–init_persistent_partitions’, ‘-D’]\r\n\r\n\r\n\x1b[01;31mWrite failed on (/dev/block/2500000.ufshci:0)\x1b[0m\r\n\r\n"b’Process Process-1:3:\r\nTraceback (most recent call last):\r\n File “/usr/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap\r\n self.run()\r\n File “/usr/lib/python3.8/multiprocessing/process.py”, line 108, in run\r\n self._target(*self._args, **self._kwargs)\r\n File “/home/rnd/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/bootburn_adb.py”, line 954, in FlashUsingADB\r\n result = self.SendFileUsingADB(partitionInfo, SkipWriteZeroChunk, TempDir, queue)\r\n File “/home/rnd/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/bootburn_adb.py”, line 493, in SendFileUsingADB\r\n AbnormalTermination(“Write failed on (%s)” % str(partitionInfo.LinuxPartitionName), nverror.NvError_FileWriteFailed)\r\n File “/home/rnd/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/flashtools_nverror.py”, line 249, in AbnormalTermination\r\n raise OSError(errorCode)\r\nOSError: 59\r\n

19

The UFS was previously provisioned.

That command also fails. Here is the log:
log2.txt (686.9 KB)

Note that:

  • I had to bind the partitions before running that command: ./make/bind_partitions -b p3710-10-a01 linux
  • Used the storage_configs/t23x/ufs-provision-p3710.cfg file instead of storage_configs/t23x/ufs-provision-256gb.cfg. That second one didn’t exist.
20

Please also provide the target side log via /dev/ttyACM0 when issuing the command. Thanks.

21

Here it is.
host.txt (686.9 KB)
target.txt (83.8 KB)

23

We are checking it and will get back to you soon. Thanks.