DriveOS 6.0.4 Flashing Failures

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Facing multiple flashing issues here. I see three issues so far:

  1. Serial ports in use (even if you’ve exited minicom, might need to reset serial ports or reboot host computer)

  2. Retry flash: Image creation during a retry will wipe previously generated files and not create them again under ~/nvidia/ folder. This is remedied by uninstalling driveos and reinstalling each time a failure occures.

3. Unresolved: “Binding of partitions failed !”

This issue seems to be different. Seems it fails after putting the board in recovery mode due to “No valid Tegra-A … must have a primary SOC” but the prior step of putting “Tegra-A on hold” and “Tegra-A in recovery” was successful.

[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [GetInfoRom(616)] : Read skuinfo from InfoRom...
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: Execute command on Aurix serial port
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [CheckFirmwareisAFW(596)] : AFW firmware found
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [setTargetConfigt23xInfoRomInfo(716)] : 940-63710-0010-D00
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [setTargetConfigt23xInfoRomInfo(717)] : None
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [setTargetConfigt23xInfoRomInfo(724)] : ******  s_InforomSkuVersion   D00
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [setTargetConfigt23xInfoRomInfo(725)] : ******  s_InforomProdInfo   940-63710-0010
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [findTargetBaseBoardName(763)] : baseBoardName found :: p3710-10-s05
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [findBaseBoardName(3440)] : Detected baseboard with default ChipSku  :: p3710-10-s05
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [updateFindBoardData(353)] : Connected Board Name details ::  baseBoard name - p3710-10-s05
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [GetTegrasAssocWithAurix(507)] : Setting Tegra-A on hold...
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [GetTegrasAssocWithAurix(509)] : Done
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [getListTargetsInRecovery(3508)] : Bus 003 Device 004: ID 0955:7045 NVIDIA Corp. Tegra On-Platform Operator
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [GetTegrasAssocWithAurix(534)] : Setting Tegra-A in recovery...
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [GetTegrasAssocWithAurix(538)] : Done
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [getListTargetsInRecovery(3508)] : Bus 003 Device 004: ID 0955:7045 NVIDIA Corp. Tegra On-Platform Operator
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [GetTegrasAssocWithAurix(548)] : No valid Tegra-A ... must have a primary SOC
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] command line used was:
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] ['/home/user/nvidia/nvidia_sdk/DRIVE_OS_6.0.4_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/bootburn.py', '--find_board_name', '-x', '/dev/ttyACM1']
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422]
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422]
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] ^[[01;31mCould not put Tegra-A in recovery^[[0m
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422]
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] [bootburn]: [__exit__(82)] : Exception in critical section :<class 'OSError'>
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422]
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] ^[[01;31mException caught in bootburn ^[[0m
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] returning to directory /home/user/nvidia/nvidia_sdk/DRIVE_OS_6.0.4_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/driveinstaller
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] Cleaning up ...
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] Cleaning temp dir
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] Traceback (most recent call last):
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] File "/home/user/nvidia/nvidia_sdk/DRIVE_OS_6.0.4_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn.py", line 288, in bootburn
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] bootburnLib.CheckRecoveryTargets()
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] File "/home/user/nvidia/nvidia_sdk/DRIVE_OS_6.0.4_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_lib.py", line 3453, in CheckRecoveryTargets
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] self.aurix.GetTegrasAssocWithAurix(self.targetConfig.s_AurixPort)
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] File "/home/user/nvidia/nvidia_sdk/DRIVE_OS_6.0.4_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/bootburn_aurix.py", line 549, in GetTegrasAssocWithAurix
[2022-09-07 12:28:10,705 root DEBUG utilities.py 294 89422] AbnormalTermination("Could not put {} in recovery".format(name), nverror.NvError_ResourceError)
[2022-09-07 12:28:10,706 root DEBUG utilities.py 294 89422] File "/home/user/nvidia/nvidia_sdk/DRIVE_OS_6.0.4_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/../bootburn_t23x_py/flashtools_nverror.py", line 249, in AbnormalTermination
[2022-09-07 12:28:10,706 root DEBUG utilities.py 294 89422] raise OSError(errorCode)
[2022-09-07 12:28:10,706 root DEBUG utilities.py 294 89422] OSError: 15
[2022-09-07 12:28:10,706 root INFO runner.py 37 89422] Failed to bind partitions!
2

Hi @eric.wu1

Did you encounter this issue during installing DRIVE OS 6.0.4 Linux SDK with SDK Manager? Please compress your ~/.nvsdm directory and provide it for our analysis.

3

Yes, this is through SDK manager. I’ve tried many times already, including reboots of host and target (including power cycle).

Specifically it looks to be this command where there are issues (I’ve tried running it manually after reading logs which gives the same result):

/home/user/nvidia/nvidia_sdk/DRIVE_OS_6.0.4_SDK_Linux_DRIVE_AGX_ORIN_DEVKITS/DRIVEOS/drive-foundation/tools/flashtools/bootburn/boot
burn.py --find_board_name -x /dev/ttyACM1

Hope it’s not an issue if I don’t include sdkm.db file. Looks like it has NV Developer user info in there.

nvsdkm.tgz (595.7 KB)

5

I’ll check it with our team and get back to you. Thanks.

6

After getting your Orin DevKit, did you only try the DRIVE OS 6.0.4 installation with SDK Manager? Please see if the second option mentioned in the below known issue helps with this issue (note: your board type is “p3710-10-s05” instead of p3710-10-a04).

image
image1122×915 101 KB

7

We had previously flashed 6.0.3 a couple times without issues. Should I still try these steps?

8

I don’t think so. But please still have a quick try. Thanks.

9

Will try when the computer is available again. Instruction you provided is how we were instructed to flash 6.0.3. SDK flashing support wasn’t available to us at the time.

In the meantime, please let me know if there’s anything else to try out.

10

Please make sure you’re running sdkmanager natively on Ubuntu 20.04 (not VM).

Please make sure to see two devices with 0955:xxxx with the below steps in two terminals.

terminal #1
lsusb

terminal #2
sudo minicom -D /dev/ttyACM1

tegrarecovery x1 on
tegrareset x1

#terminal #2
lsusb

11

@ibreakthings any update?

12

We’re running the test today. Will report back.

13

Unfortunately we couldn’t test it at all. We’re met with new issues. Serial over USB might be malfunctioning. Best case we get an error and worst case, Linux kernel on host laptop doesn’t even detect something is connected to USB. We’ve tried another Linux machine and a Windows laptop and nothing is detected. We also tried rebooting and power cycling everything.

lsusb doesn’t show any devices other than what’s on the laptop and /dev doesn’t show any ttyACM0x nodes.

Below is the most detailed error we get when something is detected. Mostly we get nothing and sometimes we just get “USB cable is bad” repeatedly.

[  558.285114] usb usb4-port4: config error
[  558.722098] ucsi_acpi USBC000:00: UCSI_GET_PDOS failed (-95)
[  562.356967] usb usb4-port4: Cannot enable. Maybe the USB cable is bad?
[  562.357059] usb usb4-port4: config error
[  566.428914] usb usb4-port4: Cannot enable. Maybe the USB cable is bad?
[  566.429145] usb usb4-port4: config error
[  570.500828] usb usb4-port4: Cannot enable. Maybe the USB cable is bad?
[  570.500911] usb usb4-port4: config error
[  573.741005] usb usb4-port4: config error
[  575.288671] ucsi_acpi USBC000:00: ucsi_handle_connector_change: GET_CONNECTOR_STATUS failed (-110)
14

Is the log snippet from your linux host system? Does it meet System Requirements? Have you reviewed DRIVE AGX Orin Developer Kit Hardware Quick Start Guide to make sure set it up correctly?

Please provide the following info (tick the boxes):
Software Version
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.8.2.10409
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

15

Yes. That’s from host system. Requirements are all validated. We were able to see /dev/ttyACMx before and had successfully flashed 6.0.3 before. My initial post in this thread noted that SDK Manager was able to see the Orin DK and was able to at least attempt to flash it. Nothing in the requirements should prevent Serial over USB from enumerating on the host. In fact, the problem is that dmesg does not show anything most of the time when USB is connected and the target is powered. This is confirmed on not just one laptop. Could this be a hardware/firmware failure at the management layer?

Software Version
DRIVE OS 6.0.3 SDK (trying to flash 6.0.4)

Target Operating System
Linux

Hardware Platform
DRIVE AGX Orin Developer Kit (not sure its number)

SDK Manager Version
other 1.8.1.10392

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager

16

I no idea now. Are you using the USB cable coming with the devkit? Please try if replacing a USB cable helps. Thanks.

17

We had already tried USB cables shipped with the kit and different cables too. Same result.

We were able to use the recovery button to put the micro USB port into recovery mode. Tried to use SDK manager and manually using the files generated by SDK manager. However, flash script still fails with same partition bind error as the initial post.

18

Have you tried rebooting the laptop? Also what is

/sys/module/usbcore/parameters/autosuspend

set to on the laptop? Please see if the below link is related to the problem you have.

19

Already mentioned we’ve tried reboots and multiple computers as the host.

As for USB autosuspend, it’s not an issue we’ve seen. It doesn’t seem to be related also based on description of that post. We have no issues with USB once the target is put into debug mode, but normal mode we have no access to ttyACM.

20

What did you mean debug mode? recovery mode? put it into recovery mode using the recovery button?
I’ll check the no /dev/ttyACMx issue with our team and get back to you.

21

I think by debug mode I meant recovery mode on the debug USB port.