Drive AGX flash error

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
2.1.0.11669
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other
VMWare (Host Windows), Ubuntu Linux 20.04 Host installed with SDK Manager

Hello,
I am experiencing flashing errors similar to many other posts here in the Forum. For example Install DRIVE OS with SDK Manager ：Failed to bind partitions - DRIVE AGX Orin / DRIVE AGX Orin General - NVIDIA Developer Forums
I followed the steps but wasnt successful.

Following the troubleshooting in the other posts:

  • I have connected all cables (original from the DevKit)
  • I can successfully connect to /dev/ttyACM0 and ACM1 with minicom,
  • I am able to see both devices with lsusb: 0955:7023 and 0955:7045.
  • I have more than 180GB free disk space and 16GB Ram

I am getting an error

Failed to bind partitions!

and in the logs

23:04:07.129 - info: NV_DRIVE_FLASH_DRIVE_COMP@DRIVE_AGX_ORIN_DEVKITS: [ Component Install Finished with Error ]
23:04:07.130 - info: NV_DRIVE_FLASH_DRIVE_COMP@DRIVE_AGX_ORIN_DEVKITS: [host] [ Disk Avail on Partition : 0.00 B ]
23:04:07.130 - info: NV_DRIVE_FLASH_DRIVE_COMP@DRIVE_AGX_ORIN_DEVKITS: [ NV_DRIVE_FLASH_DRIVE_COMP Install took 17s ]

in ~/.nvsdk/driveinstaller.log I can find the error

[2024-05-21 23:25:04,307 root DEBUG utilities.py 294 3856] [bootburn]: [findTargetBaseBoardName(763)] : baseBoardName found :: p3710-10-a01
[2024-05-21 23:25:04,307 root DEBUG utilities.py 294 3856] [bootburn]: [findBaseBoardName(3574)] : Detected baseboard with default ChipSku :: p3710-10-a01
[2024-05-21 23:25:04,307 root DEBUG utilities.py 294 3856] [bootburn]: [updateFindBoardData(354)] : Connected Board Name details :: baseBoard name - p3710-10-a01
[2024-05-21 23:25:04,307 root DEBUG utilities.py 294 3856] [bootburn]: [GetTegrasAssocWithAurix(507)] : Setting Tegra-A on hold…
[2024-05-21 23:25:04,307 root DEBUG utilities.py 294 3856] [bootburn]: [GetTegrasAssocWithAurix(509)] : Done
[2024-05-21 23:25:04,307 root DEBUG utilities.py 294 3856] [bootburn]: [getListTargetsInRecovery(3642)] : Bus 003 Device 005: ID 0955:7045 NVIDIA Corp. Tegra On-Platform Operator
[2024-05-21 23:25:04,307 root DEBUG utilities.py 294 3856] [bootburn]: [GetTegrasAssocWithAurix(534)] : Setting Tegra-A in recovery…
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] [bootburn]: [GetTegrasAssocWithAurix(538)] : Done
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] [bootburn]: [getListTargetsInRecovery(3642)] : Bus 003 Device 005: ID 0955:7045 NVIDIA Corp. Tegra On-Platform Operator
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] [bootburn]: [GetTegrasAssocWithAurix(548)] : No valid Tegra-A … must have a primary SOC
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] command line used was:
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] [‘/home/testuser/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/bootburn.py’, ‘–find_board_name’, ‘-x’, ‘/dev/ttyACM1’]
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856]
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856]
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] e[01;31mCould not put Tegra-A in recoverye[0m
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856]
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] [bootburn]: [exit(82)] : Exception in critical section :<class ‘OSError’>
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856]
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] e[01;31mException caught in bootburn e[0m
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] Traceback (most recent call last):
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] File “/home/testuser/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/bootburn.py”, line 286, in bootburn
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] bootburnLib.CheckRecoveryTargets()
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] File “/home/testuser/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/bootburn_lib.py”, line 3587, in CheckRecoveryTargets
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] self.aurix.GetTegrasAssocWithAurix(self.targetConfig.s_AurixPort)
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] File “/home/testuser/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/bootburn_aurix.py”, line 549, in GetTegrasAssocWithAurix
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] AbnormalTermination(“Could not put {} in recovery”.format(name), nverror.NvError_ResourceError)
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] File “/home/testuser/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/…/bootburn_t23x_py/flashtools_nverror.py”, line 249, in AbnormalTermination
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] raise OSError(errorCode)
[2024-05-21 23:25:04,308 root DEBUG utilities.py 294 3856] OSError: 15
[2024-05-21 23:25:04,309 root INFO runner.py 37 3856] Failed to bind partitions!
[2024-05-21 23:25:04,309 root DEBUG runner.py 39 3856] Error on line 958
[2024-05-21 23:25:04,309 root DEBUG runner.py 40 3856] Exception info: Exception Type <class ‘module.errors.FailedToBindPartitionsError’>, Traceback <traceback object at 0x7f75a74ce948>
[2024-05-21 23:25:04,309 root INFO runner.py 45 3856] driveinstaller cannot continue! Check the log file for details: /home/testuser/.nvsdkm/driveinstaller.log

At the moment I dont have a native Ubuntu 20.04 Host, so I tried with the VM instead. Could this be an issue?
What other ways to debug this issue could you propose?

Thanks a lot in advance
Best regards

3

Yes. You need to use Ubuntu 20.04.
We recommend to use docker installation method. Please use Install DRIVE OS Linux Docker Containers from NGC | NVIDIA Docs

Check using docker image
nvcr.io/drive/driveos-sdk/drive-agx-orin-linux-aarch64-sdk-build-x86:6.0.6.0-0004

Is there any reason for not upgrading to DRIVE OS 6.0.8.1?

4

Thanks for your fast reply.

It means I need Native Ubuntu 20.04 and then run Docker from within Ubuntu 20.04, correct?

Concerning upgrading the Drive OS version, I will consider as well, there is no reason actually.

Best regards

5

Correct. Please follow Install DRIVE OS 6.0.8.1 for NVIDIA Developer Users | NVIDIA Docs to flash DRIVE OS 6.0.8.1

1 Like
7

Hello,
unfortuanteley we are still experiencing the error.
In the logfiles we found more detailedly:

image
image1111×141 54.9 KB

What does it mean?

Best regards and thanks in advance.

8

Dear @Saerdna,
Could you attach the full log files? Are you using DRIVE OS 6.0.8.1 docker?

9

Hello,
At the moment I was still trying to flash with 6.0.6 and the SDK Manager, due to problems locally with my Docker.
I used a native Ubuntu 20.04 Host this time.
Could you elaborate whats the difference between the Docker way and SDK Manager? Both should probably run the same scripts to flash (bootburn etc), correct?

Please find attached the logfiles for my latest attempt with the SDK Manager.
Tomorrow I will furthermore try to flash using the Docker approach.

Kind regards and thanks a lot for your help

SDKM_logs_DRIVE_OS_6.0.6_SDK_Linux_for_DRIVE_AGX_Orin_DevKits_2024-05-24_15-36-21.zip (1.0 MB)

driveinstaller.log (1.7 MB)

10

It avoids the host platform environment related issues and noticed people moved docker does not see much flashing issues.

I notice below error in logs

b'-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_lib.py", line 3153, in FlashImages\r\n    AbnormalTermination("Flashing failed and target was not bricked as per configuration",

just want to confirm if you have modified anything in devkit? If not, lets check verifying flashing using DRIVE OS 6.0.8.1 docker as it is the latest release. BTW, what is the current DRIVE OS version on target. You can verify with cat /etc/nvidia/version-ubuntu-rootfs.txt on target. Are you able to run applications without any issue on target?

12

Thanks again for the fast feedback. I will definitely try Docker tomorrow.

The problem is, I received the System without further information or history on what has been done on it before. Currently there is no Drive OS booting up (DP output connected, but nothing shows up).
I can see the Aurix and Flash USB devices in lsusb and I can successfully connect via MiniCom Terminal Emulation.
Is there any way to get information via these two ports?

I also have no idea about modifications unfortunately.
So just guessing to exclude some possibilities: What could have been modified, i.e. could there be some kind of jumper or sth similar, which prevents flashing, that might have been set ?

Best regards

13

Do you see ubuntu login screen after connecting to /dev/ttyACM0 via minicom ? If so, you can check running cat /etc/nvidia/version-ubuntu-rootfs.txt to know DRIVE OS version
Also, please check running version on Aurix console.

Did you receive the board from any other customer or from a team member? Is the devkit opened? I see similar issue at Flashing process exited with error 116

14

I received the board from a colleague, however they are not in reach for me at the moment, hence I have little to no information really… sorry… The Devkit seems not to be opened but I cant tell if it had been opened in the past.
I am downloading the docker images atm. Will elaborate later on the status there.

I Attached the two last logs of the Minicom Terminal Connection outputs while booting. No DriveOS/Ubuntu login screen visible …

Terminal Saved Output ACM0.txt (21.8 KB)
Terminal Saved Output ACM1.txt (4.4 KB)

Thanks!

15

So, flashing with the Docker Container fails as well unfortunately.

I have Native Ubuntu 20.04 Host and followed the Docker flash instructions.

Logfiles are attached below:

log_s4euycak5hold6gjztw9mpb37021rxvq.txt (2.3 KB)
driveinstaller.log (803.5 KB)

17 
[2024-05-24 17:24:54,950 root DEBUG console_logger.py 17 47245] b'[bootburnTegra-A]: [checkMd5(425)] : \x1b[01;32m target md5 =  /dev/block/3270000.spi not found, looking by controller instance... matches host side md5 = 32ff3dcac6c57ad54d54a76f383f7e19\x1b[0m\r\n[bootburnTegra-A]: [SendFileUsingADB(608)] : Image written correctly, md5 sum matches\r\ncommand line used was:\r\n[\'/home/vector/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation//tools/flashtools/bootburn/bootburn.py\', \'-b\', \'p3710-10-a01\', \'-B\', \'qspi\', \'-x\', \'/dev/ttyACM1\', \'--init_persistent_partitions\', \'-D\']\r\n\r\n\r\n\x1b[01;31mFlashing Failed Propagated Error\x1b[0m\r\n\r\n[bootburnTegra-A]: [FlashImages(3131)] : **** FlashImages::Exception in flashing ****\r\n[bootburnTegra-A]: [FlashImages(3133)] : exception dictionary - {}\r\n[bootburnTegra-A]: [FlashImages(3142)] : flashExcept return code - 116\r\n[bootburnTegra-A]: [FlashImages(3152)] : **** s_BrickOnTCFFlashingFail is disabled in the configuration ****\r\ncommand line used was:\r\n[\'/home/vector/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation//tools/flashtools/bootburn/bootburn.py\', \'-b\', \'p3710-10-a01\', \'-B\', \'qspi\', \'-x\', \'/dev/ttyACM1\', \'--init_persistent_partitions\', \'-D\']\r\n\r\n\r\n\x1b[01;31mFlashing failed and target was not bricked as per configuration\x1b[0m\r\n\r\n\r\n\r\n\x1b[01;31mException 116 raised in bootburn_active \x1b[0m\r\nTraceback (most recent call last):\r\n  File "/home/vector/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn.py", line 156, in bootburn_active\r\n    bootburnOrin.Boot_Create_Flash_Coldboot()\r\n  File "/home/vector/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_orin.py", line 756, in Boot_Create_Flash_Coldboot\r\n    self.FlashImages(self.targetConfig.p_OutDirPath)\r\n  File "/home/vector/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive'
[2024-05-24 17:24:54,967 root DEBUG console_logger.py 17 47245] b'-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_lib.py", line 3153, in FlashImages\r\n    AbnormalTermination("Flashing failed and target was not bricked as per configuration", flashExceptRetCode)\r\n  File "/home/vector/nvidia/nvidia_sdk/DRIVE_OS_6.0.6_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/flashtools_nverror.py", line 249, in AbnormalTermination\r\n    raise OSError(errorCode)\r\nOSError: 116\r\n'
[2024-05-24 17:24:55,028 root DEBUG console_logger.py 17 47245] b'\r\n'
[2024-05-24 17:24:55,129 root ERROR pdk_flasher.py 277 47245] Flashing process exited with error 116
[2024-05-24 17:24:55,130 root INFO runner.py 37 47245] Error in flashing!
[2024-05-24 17:24:55,130 root DEBUG runner.py 39 47245] Error on line 968
[2024-05-24 17:24:55,130 root DEBUG runner.py 40 47245] Exception info: Exception Type <class 'module.errors.FlashingError'>, Traceback <traceback object at 0x7f594ab60a88>

[2024-05-29 09:52:28,469 root DEBUG console_logger.py 17 21] b"command line used was:\r\n['/drive/drive-foundation//tools/flashtools/bootburn/bootburn.py', '-b', 'p3710-10-a01', '-B', 'qspi', '-x', '/dev/ttyACM1']\r\n\r\n\r\n\x1b[01;31mFlashing Failed Propagated Error\x1b[0m\r\n\r\n"
[2024-05-29 09:52:28,471 root DEBUG console_logger.py 17 21] b'[bootburnTegra-A]: [FlashImages(3440)] : **** FlashImages::Exception in flashing ****\r\n'
[2024-05-29 09:52:28,472 root DEBUG console_logger.py 17 21] b'[bootburnTegra-A]: [FlashImages(3442)] : exception dictionary - {}\r\n'
[2024-05-29 09:52:28,473 root DEBUG console_logger.py 17 21] b'[bootburnTegra-A]: [FlashImages(3451)] : flashExcept return code - 117\r\n'
[2024-05-29 09:52:28,474 root DEBUG console_logger.py 17 21] b"[bootburnTegra-A]: [FlashImages(3461)] : **** s_BrickOnTCFFlashingFail is disabled in the configuration ****\r\ncommand line used was:\r\n['/drive/drive-foundation//tools/flashtools/bootburn/bootburn.py', '-b', 'p3710-10-a01', '-B', 'qspi', '-x', '/dev/ttyACM1']\r\n\r\n\r\n\x1b[01;31mFlashing failed and target was not bricked as per configuration\x1b[0m\r\n\r\n\r\n\r\n\x1b[01;31mException 117 raised in bootburn_active \x1b[0m\r\n"
[2024-05-29 09:52:28,475 root DEBUG console_logger.py 17 21] b'Traceback (most recent call last):\r\n  File "/drive/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn.py", line 156, in bootburn_active\r\n    bootburnOrin.Boot_Create_Flash_Coldboot()\r\n  File "/drive/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_orin.py", line 760, in Boot_Create_Flash_Coldboot\r\n    self.FlashImages(self.targetConfig.p_OutDirPath)\r\n  File "/drive/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_lib.py", line 3462, in FlashImages\r\n    AbnormalTermination("Flashing failed and target was not bricked as per configuration", flashExceptRetCode)\r\n  File "/drive/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/flashtools_nverror.py", line 251, in AbnormalTermination\r\n    raise OSError(errorCode)\r\nOSError: 117\r\n'
[2024-05-29 09:52:30,548 root DEBUG console_logger.py 17 21] b'\r\n'
[2024-05-29 09:52:30,649 root ERROR pdk_flasher.py 280 21] Flashing process exited with error 117
[2024-05-29 09:52:30,649 root INFO runner.py 37 21] Error in flashing!
[2024-05-29 09:52:30,650 root DEBUG runner.py 39 21] Error on line 990
[2024-05-29 09:52:30,650 root DEBUG runner.py 40 21] Exception info: Exception Type <class 'module.errors.FlashingError'>, Traceback <traceback object at 0x7fba1b5d3600>

Such error messages haven’t been seen except in Flashing process exited with error 116 due to a component change.

Could you please get the current version by running the following command in the Aurix console?
NvShell>version

Power cycling both the host and target system is always worth a try.

Additionally, could you check with your colleague to get the previous status of this system? Also, what is the serial number of the system?

18

Output of

NvShell> version
DRIVE-V6.0.0-P3710-AFW-Aurix-StepB-5.00.02

I also manually tried (similar to your solution from Flash failed. Orin no more recoverable or flashable - #12 by SivaRamaKrishnaNV) :

root@6.0.8.1-0006-build-linux-sdk:/drive# sudo -E bootburn.py -b p3710-10-a01 -B qspi -x /dev/ttyACM1

and the result is

[bootburnTegra-A]: [FlashUsingADB(889)] : [2500000.ufshci:0] Flashing for Partition A_secure-os
…
Write failed on (/dev/block/2500000.ufshci:0)
…

Therefore there seem to be problems with the flashing of that Flash connected via ufshci ?
As can also be seen in the bootlog file above,

Some issues with UFS can also be detected:

E> Top caller module: UFS, error module: UFS, reason: 0x04, aux_info: 0x00

Are these related?
The other partitions eMMC seem to flash properly.

One more Edit:
Now finally, via ACM0 sometimes there is bash shell, but it doesn’t seem to be be a full Ubuntu install… atm I can not reproduce how to boot into the shell. Will give more Infos later.

Thanks for your fast feedback up to now!
Best regards

19

Could you follow Setting Up DRIVE OS Linux with NVIDIA GPU Cloud (NGC) and attempt to upgrade to 6.0.4?
In Valid Migration Paths, there is no support for upgrading from 6.0.0 to any version. though.

1 Like
20

Dear @Saerdna,
Could you provide any update? Can we close the topic?

21

Hello,
The solution was to provision the UFS device using the bootburn script, as it is described in the PDK Developer guide. After the provisioning, a direct upgrade to 6.0.8.1 was possible.

Therefore, we can close the topic now. Thanks again for your help !

Best regards

23

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.