Drive AGX flashing failed

Hi,

I am trying to flash DriveOS 9.0 to Drive AGX Development Kit, but the following error occurs:

...
Failed to get Tegra instance.
==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/ta_on <==

==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/tb_on <==

==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/tb_off <==

 ------------ Stack Trace ------------
stack frame 0 - 297 AbnormalTermination /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/bootburn_lib.sh
stack frame 1 - 480 GetTegrasAssocWithAurix /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh
stack frame 2 - 509 CheckRecoveryTargets /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh
stack frame 3 - 946 main /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh
-------------------------------------

error-target-instance

Flashing process exited with error 15
Error in flashing!
...

Host system:
Ubuntu 16.04 LTS (Kernel version 4.15.0-46-generic) with Nvidia SDK Manager 1.0.0.5517

Drive AGX board version:
E3550: 699-63550-0001-501 G1
T194-A: REV A02 BR:03
T194-B: REV A02 BR:03

Aurix firmware:
DRIVE-V5.1.0-E3550-EB-Aurix-With3LSS-3.02.07.hex
DRIVE-V5.1.0-E3550-NV-Aurix-UPDATE-StepA-1.29.00.hex

Could you please help?

Thanks
Rupert
agx_flashing_issue_log.zip (1.64 MB)

Please find the log files (as suggested in DRIVE AGX Developer Kit Flashing Technical Bulletin for DriveOS 9.0) attached in the previous posting.

Hi rsn,

Please try with the workaround mentioned in https://devtalk.nvidia.com/default/topic/1070169/general/flashing-error-could-not-execute-command-tegrareset-x1-h/post/5421889/#5421889 and see if it helps on your case. Thanks!

Hi,

I increased “l_readAttempts” to 10000, but still the same error occurs:

[2020-02-12 19:51:40,836 root DEBUG console_logger.py 17 22378] b'Done\r\n'
[2020-02-12 19:51:40,935 root DEBUG console_logger.py 17 22378] b'Enabling SIGINT <Ctrl+C>\r\n'
[2020-02-12 19:51:40,940 root DEBUG console_logger.py 17 22378] b'Failed to get Tegra instance.\r\n'
[2020-02-12 19:51:40,941 root DEBUG console_logger.py 17 22378] b'==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/ta_on <==\r\n\r\n==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/tb_on <==\r\n\r\n==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/tb_off <==\r\n'
[2020-02-12 19:51:40,942 root DEBUG console_logger.py 17 22378] b'\r\n ------------ Stack Trace ------------\r\n'
[2020-02-12 19:51:40,943 root DEBUG console_logger.py 17 22378] b'stack frame 0 - 297 AbnormalTermination /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/bootburn_lib.sh\r\n'
[2020-02-12 19:51:40,943 root DEBUG console_logger.py 17 22378] b'stack frame 1 - 480 GetTegrasAssocWithAurix /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh\r\n'
[2020-02-12 19:51:40,944 root DEBUG console_logger.py 17 22378] b'stack frame 2 - 509 CheckRecoveryTargets /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh\r\n'
[2020-02-12 19:51:40,944 root DEBUG console_logger.py 17 22378] b'stack frame 3 - 946 main /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh\r\n'
[2020-02-12 19:51:40,945 root DEBUG console_logger.py 17 22378] b'-------------------------------------\r\n\r\n'
[2020-02-12 19:51:40,945 root DEBUG console_logger.py 17 22378] b'error-target-instance\r\n'
[2020-02-12 19:51:41,029 root DEBUG console_logger.py 17 22378] b'\r\n'
[2020-02-12 19:51:41,130 root ERROR pdk_flasher.py 168 22378] Flashing process exited with error 15
[2020-02-12 19:51:41,130 root CRITICAL runner.py 37 22378] Error in flashing!
[2020-02-12 19:51:41,131 root DEBUG runner.py 38 22378] Exception info:
Traceback (most recent call last):
  File "./pdkinstaller", line 777, in <module>
    pdk_flasher.flash()
  File "/home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/pdkinstaller/module/pdk_flasher.py", line 171, in flash
    raise FlashingError
module.errors.FlashingError: Error in flashing!
[2020-02-12 19:51:41,132 root CRITICAL runner.py 43 22378] pdkinstaller cannot continue! Check the log file for details: /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/pdkinstaller/pdkinstaller.log

Please give us more background of this dev kit. Have you ever installed DRIVE Software 9.0 successfully? Wy don’t you install DRIVE Software 10.0? Have you ever installed DRIVE Software 10.0 on it?

No, it is a brand-new board and I never installed Drive Software 9.0 successfully (and it is the only software I have ever tried to install on this board). I have to stick to version 9.0 to be consistent within the project with other colleagues.

BR
rsn

Hi rupert.schorn,

I’ll check internally and get back to you here.

Hi rupert.schorn,

just to make sure, are you able to open a minicom connection to Aurix?
and are you able to execute the command “tegrarecovery x1 on” ?

Hi,

yes I am able to connect to the Aurix and execute the “tegrarecovery x1 on” command:

shell> tegrarecovery x1 on
Info: Executing cmd: tegrarecovery, argc: 2, args: x1 on
Command Executed
shell>

BR
rsn

I assume you modified ~/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/bootburn.sh and then re-ran SDK Manager to install.
Could you try with changing “sleep 1” to “sleep 5” instead of changing l_readAttempts?

function ExecuteAurixCommand()
{
...
    local l_readAttempts=10
    local l_checkError=                                                                                                                                                                                            
    local l_checkSuccess=

    CheckAurixExclusiveAccess $l_Aurix
    trap "" SIGINT

    TraceNext "Dump Aurix serial port output to destination"
    sudo python3 ${p_PyFlashPath}/misc/aurixSerial.py  "$l_Aurix"  "$p_FlashFiles/$l_AurixLogFile" &
    l_AurixPidParent=$!
    disown ${l_AurixPidParent}
    <b>sleep 1</b>

Do you still see “Process CMD (14840) still running. Trying to kill after 5 seconds” messages as below?

Running flash command: sudo -E /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh -b e3550b03-t194a -B qspi
-x /dev/ttyUSB3
[host] [sudo] password for schorn: ^M
Successfully acquired lock over /var/lock/LCK..bootburn^M
Successfully acquired lock over /var/lock/LCK..ttyUSB3^M
Read skuinfo from InfoRom...^M
<b>Process CMD (14840) still running. Trying to kill after 5 seconds</b>^M
Disabling SIGINT <Ctrl+C> temporarily^M
Setting Tegra-A on hold... Process CMD (14901) still running. Trying to kill after 5 seconds^M
Done^M
Setting Tegra-B on hold... Process CMD (14933) still running. Trying to kill after 5 seconds^M
Done^M
Setting Tegra-A in recovery... Process CMD (14969) still running. Trying to kill after 5 seconds^M
Process CMD (15000) still running. Trying to kill after 5 seconds^M
Done^M
Setting Tegra-B in recovery... Process CMD (15052) still running. Trying to kill after 5 seconds^M
Process CMD (15088) still running. Trying to kill after 5 seconds^M
Done^M
Enabling SIGINT <Ctrl+C>^M
Failed to get Tegra instance.^M                                                                                                                                                                                    
==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/ta_on <==^M
^M
==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/tb_on <==^M
^M
==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/tb_off <==^M
^M
 ------------ Stack Trace ------------^M
stack frame 0 - 297 AbnormalTermination /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/bootburn_lib.sh^M
stack frame 1 - 480 GetTegrasAssocWithAurix /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh^M
stack frame 2 - 509 CheckRecoveryTargets /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh^M
stack frame 3 - 946 main /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh^M
-------------------------------------^M
^M
/Failed to get Tegra instance.

BTW, may I know if with the default image the dev kit can boot up successfully?

Hi,

yes, the boot kit boots up successfully with the default image.

Increasing “sleep 1” to “sleep 5” did not help, still the same error occurs:

[2020-02-13 16:48:15,093 root DEBUG console_logger.py 17 4283] b'\r\n'
[2020-02-13 16:48:15,131 root DEBUG console_logger.py 17 4283] b'Successfully acquired lock over /var/lock/LCK..bootburn\r\n'
[2020-02-13 16:48:15,131 root DEBUG console_logger.py 17 4283] b'Successfully acquired lock over /var/lock/LCK..ttyUSB3\r\n'
[2020-02-13 16:48:15,163 root DEBUG console_logger.py 17 4283] b'Read skuinfo from InfoRom...\r\n'
[2020-02-13 16:48:18,243 root DEBUG console_logger.py 17 4283] b'Process CMD (5137) still running. Trying to kill after 5 seconds\r\n'
[2020-02-13 16:48:23,839 root DEBUG console_logger.py 17 4283] b'Disabling SIGINT <Ctrl+C> temporarily\r\nSetting Tegra-A on hold... '
[2020-02-13 16:48:34,022 root DEBUG console_logger.py 17 4283] b'Process CMD (5198) still running. Trying to kill after 5 seconds\r\n'
[2020-02-13 16:48:39,526 root DEBUG console_logger.py 17 4283] b'Done\r\nSetting Tegra-B on hold... '
[2020-02-13 16:48:49,703 root DEBUG console_logger.py 17 4283] b'Process CMD (5229) still running. Trying to kill after 5 seconds\r\n'
[2020-02-13 16:48:55,207 root DEBUG console_logger.py 17 4283] b'Done\r\n'
[2020-02-13 16:48:55,300 root DEBUG console_logger.py 17 4283] b'Setting Tegra-A in recovery... '
[2020-02-13 16:49:05,473 root DEBUG console_logger.py 17 4283] b'Process CMD (5269) still running. Trying to kill after 5 seconds\r\n'
[2020-02-13 16:49:21,145 root DEBUG console_logger.py 17 4283] b'Process CMD (5303) still running. Trying to kill after 5 seconds\r\n'
[2020-02-13 16:49:26,650 root DEBUG console_logger.py 17 4283] b'Done\r\n'
[2020-02-13 16:49:26,743 root DEBUG console_logger.py 17 4283] b'Setting Tegra-B in recovery... '
[2020-02-13 16:49:36,937 root DEBUG console_logger.py 17 4283] b'Process CMD (5355) still running. Trying to kill after 5 seconds\r\n'
[2020-02-13 16:49:52,627 root DEBUG console_logger.py 17 4283] b'Process CMD (5386) still running. Trying to kill after 5 seconds\r\n'
[2020-02-13 16:49:58,131 root DEBUG console_logger.py 17 4283] b'Done\r\n'
[2020-02-13 16:49:58,234 root DEBUG console_logger.py 17 4283] b'Enabling SIGINT <Ctrl+C>\r\n'
[2020-02-13 16:49:58,239 root DEBUG console_logger.py 17 4283] b'Failed to get Tegra instance.\r\n'
[2020-02-13 16:49:58,239 root DEBUG console_logger.py 17 4283] b'==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/ta_on <==\r\n\r\n==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/tb_on <==\r\n\r\n==> /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/_temp_dump/tb_off <==\r\n'
[2020-02-13 16:49:58,240 root DEBUG console_logger.py 17 4283] b'\r\n ------------ Stack Trace ------------\r\nstack frame 0 - 297 AbnormalTermination /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x/bootburn_lib.sh\r\n'
[2020-02-13 16:49:58,241 root DEBUG console_logger.py 17 4283] b'stack frame 1 - 480 GetTegrasAssocWithAurix /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh\r\n'
[2020-02-13 16:49:58,242 root DEBUG console_logger.py 17 4283] b'stack frame 2 - 509 CheckRecoveryTargets /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh\r\n'
[2020-02-13 16:49:58,243 root DEBUG console_logger.py 17 4283] b'stack frame 3 - 946 main /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh\r\n'
[2020-02-13 16:49:58,243 root DEBUG console_logger.py 17 4283] b'-------------------------------------\r\n\r\n'
[2020-02-13 16:49:58,244 root DEBUG console_logger.py 17 4283] b'error-target-instance\r\n'
[2020-02-13 16:49:58,331 root DEBUG console_logger.py 17 4283] b'\r\n'
[2020-02-13 16:49:58,432 root ERROR pdk_flasher.py 168 4283] Flashing process exited with error 15
[2020-02-13 16:49:58,432 root CRITICAL runner.py 37 4283] Error in flashing!
[2020-02-13 16:49:58,432 root DEBUG runner.py 38 4283] Exception info:
Traceback (most recent call last):
  File "./pdkinstaller", line 777, in <module>
    pdk_flasher.flash()
  File "/home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/pdkinstaller/module/pdk_flasher.py", line 171, in flash
    raise FlashingError
module.errors.FlashingError: Error in flashing!
[2020-02-13 16:49:58,435 root CRITICAL runner.py 43 4283] pdkinstaller cannot continue! Check the log file for details: /home/schorn/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/pdkinstaller/pdkinstaller.log
[2020-02-13 16:49:58,435 root DEBUG runner.py 60 4283] Cleaning up...
[2020-02-13 16:49:58,435 root DEBUG utilities.py 124 4283] Deleting temporary directories...
[2020-02-13 16:49:58,435 root DEBUG utilities.py 126 4283] Deleting /tmp/pdkinstall-tmp-stqjmc3t-bind
[2020-02-13 16:49:58,435 root DEBUG runner.py 62 4283] Closing the log file.

Message “Process CMD (xxxxx) still running. Trying to kill after 5 seconds” is still there…

Could you help associate the usb bus:device with the Tegra Instance. To do this we open up a serial communication to Aurix (/dev/ttyUSB3) and then execute the commands:

tegrarecovery x1 on
tegrareset x1
tegrarecovery x2 on
tegrareset x2

In another terminal we do the following:

lsusb -d 0955:

and see something similar to

Bus 002 Device 116: ID 0955:7019 NVidia Corp.
Bus 002 Device 117: ID 0955:7019 NVidia Corp.

Does everything look normal on your side? Thanks!

Output looks normal:

Tegra recovery and reset commands:

shell> tegrarecovery x1 on
Info: Executing cmd: tegrarecovery, argc: 2, args: x1 on
Command Executed
shell> tegrareset x1
Info: Executing cmd: tegrareset, argc: 1, args: x1
shell> Command Executed

shell> tegrarecovery x2 on
Info: Executing cmd: tegrarecovery, argc: 2, args: x2 on
Command Executed
shell> tegrareset x2
Info: Executing cmd: tegrareset, argc: 1, args: x2
shell> Command Executed

shell>

lsusb command:

schorn@OptiPlex-990:~$ lsusb -d 0955:
Bus 002 Device 029: ID 0955:7019 NVIDIA Corp.
Bus 002 Device 028: ID 0955:7019 NVIDIA Corp.
schorn@OptiPlex-990:~$

We are still discussing how to further clarify your issue internally.

At the same time, could you have a try of installing DRIVE Software 10.0 to see if the same issue? If yes, we can also check people’s experience in DRIVE Software 10.0. If not, we can focus on related fix in DRIVE Software 10.0. BTW, installing DRIVE Software 10.0 needs ubuntu 18.04 host.

also would like to know if the SDK Manager ran on virtualization solftware or on ubuntu 16.04 natively?

SDK manager ran on ubuntu 16.04 natively.

Hi,

installing Drive SW 10 was successful.

Thanks for providing the result of DRIVE Software 10.0 installation!

For DRIVE Software 9.0 installation, could you remove ,"-x %s" of related/all board configurations in ~/nvidia/nvidia_sdk/DRIVE_Software_9.0_Linux_hyperion_E3550/DriveSDK/pdkinstaller/manifests/flash_pdk_config/flash_config.json (as below) and then install with SDK Manager? Thanks!

"board": {
  "e3550b01-t194a": {
    "path": "/tools/host/flashtools/bootburn_t19x/",
    "scripts_home": "./scripts/",
    "identify_aurix": "identifyAurix.sh",
    "args": [
      "-b e3550b01-t194a",
      "-B qspi",
      "-x %s"

    ]
  },


}

Can I install DRIVE Software 9.0 if version 10.0 is already installed?

Can I install DRIVE Software 9.0 with Ubuntu 18.04 on the host machine?