Not able to boot and reflash Drive AGX Xavier

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.8.0.10363
other

Host Machine Version
native Ubuntu 18.04
other

Hi There,

Here we have situation with xavier for long time.

Hardware: T194A/B

Main issue:
Xavier was working with SW9.0, and working well, but it was crashed now, problem as below:

  1. Xavier A not able to book, we can reflash the xaiver with ubuntu16 and SW9.0 environment (upgrade the python to 3.7 with pip 3.7) to reflash it with option to reflash xavier A, output is still xaiver A desktop picture from HDMI XavierB. (this is because ttyUSB2 is not accessable, while ttyUSB3, ttyUSB6 are all accessable)

  2. Then we tried to reflash the aurix with windows infineon software , seems it works, the firmware we were using as below:
    DRIVE-V5.1.0-E3550-NV-Aurix-IFW-StepA-1.29.00
    DRIVE-V5.1.0-E3550-NV-Aurix-UPDATE-StepA-1.29.00
    DRIVE-V5.1.0-E3550-EB-Aurix-With3LSS-3.02.07
    DRIVE-V5.1.0-P3479-EB-Aurix-With3LSS-3.02.07
    DRIVE-V5.1.0-P3479-NV-Aurix-UPDATE-StepA-1.29.00

but could not resolve the problem.

we tried to reflash the different version including 10.0, 5.2.6, seems was successfully, but was only for XavierB as well, xavier A always not able to book.

  1. Now recently all the version reflash failed, we can’t access all the USB port including ttyUSB 2.3.6, but with ubuntu 18, seems when reflash 5.2.6, USB3 is still accessable (not sure, will post some information later).

And also always when you poweroff the xavier, you can’t poweron same day, always waiting for the next day to poweron it (poweron here means fan works, LED on board flashing continuely), minutes later, the IP from router terminal list will dispear (seems it’s dead again), but before that, may able to access via minicom -D /dev/ttyUSB3.

  1. currently SW9.0 with ubuntu 16 log as below:
6:47:30 DEBUG: running command < true >
16:47:30 INFO: command finished successfully
16:47:30 DEBUG: running command < stat /dev/ttyUSB3 >
16:47:30 INFO: command finished successfully
16:47:30 DEBUG: running command < true >
16:47:30 INFO: command finished successfully
16:47:30 DEBUG: running command < lsof /dev/ttyUSB3 2>/dev/null >
16:47:31 DEBUG: command terminated with error
16:47:31 DEBUG: running command < true >
16:47:31 INFO: command finished successfully
16:47:31 DEBUG: Flash Xavier A - flash: using adapter to install NV_FLASH_XAVIER_PDKFLASH_A_COMP@DDPX to /media/jason/data/NVIDIA-SW9/DW9SDK/DRIVE_Software_9.0_Linux_hyperion_DRIVE_AGX_XAVIER
16:47:31 INFO: Flash Xavier A - flash: Installing component 'Flash Xavier A' (NV_FLASH_XAVIER_PDKFLASH_A_COMP)
16:47:31 INFO: Flash Xavier A - flash: exec_command: cd pdkinstaller && ./bin/python3 -E ./pdkinstaller --installtype bindflash --boardtype ES --pdkinstalledpath /media/jason/data/NVIDIA-SW9/DW9SDK/DRIVE_Software_9.0_Linux_hyperion_DRIVE_AGX_XAVIER/DriveSDK --configjson ./manifests/install_pdk_config/av_ddp_pegasus_hyp71_linux_tegraA_install_config.json --consoleout enable
16:49:32 INFO: Flash Xavier A - flash: Password for jason:
16:49:32 ERROR: Flash Xavier A - flash: command terminated with error
16:49:32 SUMMARY: Flash Xavier A - flash: First Error: Installation failed.

we changed the 2nd power supply, issue is similar, seems it’s not power supply issue. voltage is as below

BTW, once we can access the xavier via putty terminal, always USB14 is accesable, not USB4, minicom configuration is not sure in right status or not.

appreciate if anyone can help to solve the issue.

Here attache the xavier flash with Ubuntu18.04+Software 5.2.6 update:

  1. Flash Xavier B individually works well as before, but not able to be connected with network (no devices )
  2. Flash Xavier A or Flash both A+B failed.

below is the record of flashing A

22:14:36 INFO: Flash Xavier A - flash: NV_WORKSPACE:NV_TARGETUSER:NV_TARGETPWD:NV_EULA_ACCEPTED:
22:14:36 INFO: Flash Xavier A - flash: EULA has been accepted!
22:14:36 INFO: Flash Xavier A - flash: [host] [sudo] password for datalab:
22:14:36 INFO: Flash Xavier A - flash:
22:14:36 INFO: Flash Xavier A - flash: Executing /opt/nvidia/driveos/5.2.6.0/filesystems/build-kit/bin/build_kit.py for user account setup.
22:14:36 INFO: Flash Xavier A - flash: Reading Configuration File: '/opt/nvidia/driveos/5.2.6.0/filesystems//build-kit/build-kit.config'
22:14:36 INFO: Flash Xavier A - flash: Executing mount
22:14:36 INFO: Flash Xavier A - flash: Executing grep "binfmt_misc on /proc/sys/fs/binfmt_misc"
22:14:36 INFO: Flash Xavier A - flash: Executing update-binfmts --enable qemu-aarch64
22:14:36 INFO: Flash Xavier A - flash: update-binfmts: warning: qemu-aarch64 already enabled in kernel.
22:14:36 INFO: Flash Xavier A - flash: Using default base filesystem mirrors.
22:14:36 INFO: Flash Xavier A - flash: No debian packages requested in CONFIG. Skipping MANIFEST generation.
22:14:36 INFO: Flash Xavier A - flash: No debian packages requested in MANIFEST. Skipping package installs.
22:14:36 INFO: Flash Xavier A - flash: Executing chroot /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs grep -q -E ":1000:" /etc/passwd
22:14:36 INFO: Flash Xavier A - flash: Executing chroot /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs grep -E ":1000:" /etc/passwd
22:14:36 INFO: Flash Xavier A - flash: Executing chroot /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs usermod -s /bin/bash -l nvidia -m -d /home/nvidia nvidia
22:14:36 INFO: Flash Xavier A - flash: usermod: no changes
22:14:36 INFO: Flash Xavier A - flash: Executing chroot /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs grep -q -E "^nvidia:" /etc/group
22:14:36 INFO: Flash Xavier A - flash: Executing chroot /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs groupmod -n nvidia nvidia
22:14:36 INFO: Flash Xavier A - flash: Executing chroot /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs dpkg-query -Wf '${Package}=${Version}
22:14:36 INFO: Flash Xavier A - flash: '
22:14:36 INFO: Flash Xavier A - flash: Executing chroot /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs dpkg -l
22:14:36 INFO: Flash Xavier A - flash: =======Build Kit Successfully Finished Execution=======
22:14:36 INFO: Flash Xavier A - flash:
22:14:36 INFO: Flash Xavier A - flash: Executing Cleanup Routine for Linux Build-Kit on Exit.
22:14:36 INFO: Flash Xavier A - flash:
22:14:36 INFO: Flash Xavier A - flash: Executing umount /tmp/tmpjnqz80or//rootfs_mount/
22:14:36 INFO: Flash Xavier A - flash: umount: /tmp/tmpjnqz80or//rootfs_mount/: not mounted.
22:14:36 INFO: Flash Xavier A - flash:
22:14:36 INFO: Flash Xavier A - flash: Executing Cleanup Routine for Build-Kit on Exit.
22:14:36 INFO: Flash Xavier A - flash:
22:14:36 INFO: Flash Xavier A - flash: umount: /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs/sys: not mounted.
22:14:36 INFO: Flash Xavier A - flash: umount: /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS//drive-t186ref-linux/targetfs/proc: not mounted.
22:14:36 INFO: Flash Xavier A - flash: BOARD SKU : 940-63550-2000-100
22:14:36 INFO: Flash Xavier A - flash: Board PCB revision is b03
22:14:36 INFO: Flash Xavier A - flash: Running flash command: sudo -E /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh -b e3550b03-t194a -B qspi -x /dev/ttyUSB3
22:14:36 INFO: Flash Xavier A - flash: [host] [sudo] password for datalab:
22:14:36 INFO: Flash Xavier A - flash: FlashCommandLineFromUser : -b e3550b03-t194a -B qspi -x /dev/ttyUSB3
22:14:36 INFO: Flash Xavier A - flash: Using Python scripts
22:14:36 INFO: Flash Xavier A - flash: /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh: ./bootburn.py: /usr/bin/python: bad interpreter: No such file or directory
22:14:36 INFO: Flash Xavier A - flash:
22:14:36 INFO: Flash Xavier A - flash: Flashing process exited with error 126
22:14:36 INFO: Flash Xavier A - flash: Error in flashing!
22:14:36 INFO: Flash Xavier A - flash: pdkinstaller cannot continue! Check the log file for details: /home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS/pdkinstaller/pdkinstaller.log
22:14:36 INFO: Flash Xavier A - flash: [ Package Install Finished with Error ]
22:14:36 INFO: Flash Xavier A - flash: [host] [ 57.61 MB used. Disk Avail: 405.58 GB ]
22:14:36 INFO: Flash Xavier A - flash: [ NV_DRIVE_FLASH_DDPX_X1_COMP Install took 21s ]
22:14:36 ERROR: Flash Xavier A - flash: command terminated with error
22:14:36 SUMMARY: Flash Xavier A - flash: First Error: Installation failed.

lsusb @host

Bus 002 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 010: ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
Bus 001 Device 009: ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
Bus 001 Device 008: ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
Bus 001 Device 006: ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
Bus 001 Device 004: ID 0424:2807 Standard Microsystems Corp. 
Bus 001 Device 003: ID 046d:c31c Logitech, Inc. Keyboard K120
Bus 001 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 001 Device 007: ID 8087:0025 Intel Corp. 
Bus 001 Device 005: ID 413c:301a Dell Computer Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub


Tonight we tried again to flash with 5.2.6 and succeed again for XavierB, failed again for XavierA, voltages showing different with the last access via aurix shell, also xavier B can reboot in 20 mins after power off and working stable (but some issue such as can’t install gdm3 in xavier B via ssh).

Xavier A not accessible.

Shell>showvoltages
INPUTS:
  KL30_VBAT = 12.097 V
  KL30_POWER = 12.075 V
  KL15_POWER = 12.133 V
AURIX:
  AURIX_1V3 = 1.251 V
  AURIX_3V3 = 3.296 V
  AURIX_5V = 4.991 V
  CAN1_5V = 5.003 V
  CAN2_5V = 4.996 V
  FR_5V = 5.009 V
  HRNS_GPI = 0.411 V
SYSTEM:
  VBAT_SYS = 12.075 V
  VBATSYS_ISENSE = 3.042 A
  SYS_5V = 5.012 V
  SYS_0V85 = 0.847 V
  SYS_0V92_1 = 0.951 V
  SYS_0V92_2 = 0.908 V
  SYS_1V0 = 0.981 V
  SYS_1V1_2 = 1.100 V
  SYS_1V2 = 1.210 V
  SYS_1V5_2 = 1.512 V
  SYS_1V8_1 = 1.824 V
  SYS_1V8_2 = 1.802 V
  SYS_2V1 = 2.128 V
  SYS_2V5 = 2.523 V
  SYS_3V3_1 = 3.336 V
  SYS_3V3_2 = 3.306 V
  CE_PREREG = 0.011 V
Tegra A:
  VBAT_TEG = 12.082 V
  VBATTEG_ISENSE = 1.958 A
  XA_PREREG = 11.891 V
  XA_5V = 5.098 V
  XA_5V_SW = 5.090 V
  XA_VDD_1V0 = 1.002 V
  XA_VDD_1V8_AO = 1.794 V
  XA_VDD_1V8_HS = 1.807 V
  XA_VDD_1V8_LS = 1.805 V
  XA_VDD_CPU = 1.102 V
  XA_VDD_CV = 1.094 V
  XA_VDD_DDR2 = 1.118 V
  XA_VDD_DDRQ = 0.608 V
  XA_VDD_DDR_1V1 = 1.111 V
  XA_VDD_GPU = 1.107 V
  XA_VDD_SOC = 0.002 V
  X1_DGPU_THERM_ALERT_N = 1.788 V
  VBAT_SXMA = 0.014 V
  VBATSXMA_ISENSE = 0.000 A
  SXMA_PREREG = 0.047 V
Tegra B:
  XB_PREREG = 11.933 V
  XB_5V = 5.147 V
  XB_5V_SW = 5.129 V
  XB_VDD_1V0 = 1.003 V
  XB_VDD_1V8_AO = 1.798 V
  XB_VDD_1V8_HS = 1.804 V
  XB_VDD_1V8_LS = 1.800 V
  XB_VDD_CPU = 0.832 V
  XB_VDD_CV = 0.915 V
  XB_VDD_DDR2 = 1.108 V
  XB_VDD_DDRQ = 0.608 V
  XB_VDD_DDR_1V1 = 1.123 V
  XB_VDD_GPU = 0.874 V
  XB_VDD_SOC = 0.855 V
  X2_DGPU_THERM_ALERT_N = 1.786 V
  VBAT_SXMB = 12.082 V
  VBATSXMB_ISENSE = 0.040 A
  SXMB_PREREG = 12.017 V
CVM:
  P1_PREREG = 11.915 V
  P1_5V = 5.079 V


Thanks if anyone can help.

BTW. I think as we reflashed with 5.2.6, could be not able to flash with SW9.0 again directly even if we could fix the boot issue of xavier A?

And, is possible hardware broken?

Some additional information from aurix shell:
with both power supply, same issue happens, aurixrest and poweron x1 will have below feedback: (poweron without x1/x2 works, but once again, all failed, also unplug the power supply for harlf hour, still failed, guess back to original status to wait for over night for XavierB boot again.)

Shell>aurixreset 
 CIM powered off
Power off the system
Command Executed


DDPX Aurix Serial Console
E3550-B03
with TLF35584 B/C-Step
SW Version  1.29.00
Debug Version, SW BUILD 7
TC397 Step A

Enter 'help' to see the available commands

Shell>System_PowerOn: Reading PG status: Timeout Error!
X1_XA_VDD_SOC_XA_VDD_DDR2_XA_VDD_DDRQ_PG
Set low threshold: 5600 mv, raw value 0x2fd 

Shell>
Shell>status
Alive: 00:00:30
CPU0 Frequency: 300000000 Hz
STM0 Frequency: 100000000 Hz
CPU1 Frequency: 300000000 Hz
STM1 Frequency: 100000000 Hz
CPU2 Frequency: 300000000 Hz
STM2 Frequency: 100000000 Hz
Shell>poweron x1
System_PowerOn: Reading PG status: Timeout Error!
X1_XA_VDD_SOC_XA_VDD_DDR2_XA_VDD_DDRQ_PG
Command Fail 
Shell>poweron x1
System_PowerOn: Reading PG status: Timeout Error!
X1_XA_VDD_SOC_XA_VDD_DDR2_XA_VDD_DDRQ_PG
Command Fail 
Shell>FAN RPM reading error: voerflow flag is set
Shell>poweron   
Detect 2nd Harness is not connected.
If you are using DGPU, please contact your hardware representative to get 2nd he
Configured BootStraps for X1 as QSPI
Configured BootStraps for X2 as QSPI
System Powering on in CE_MODE = 0 ...
Init SJA1105 sucessfully
Read valid mac address from I2C ROM
MAC Address programmed to this device: 00:04:4b:cb:bf:79
Set IP address to 10.42.0.146
Initial 88Q2112
Initial 88Q2112 A0 silicon slave address 1
Initial 88Q2112 A0 silicon slave address 2
Initial 88Q2112 A0 silicon slave address 3
Initial 88Q2112 A0 silicon slave address 4
Initial 88Q2112 A0 silicon slave address 5
Initial 88Q2112 A0 silicon slave address 6
Initial 88Q2112 A0 silicon slave address 7
88E6321_1 is in CPU mode
88E6321_2 is in CPU mode
Initial 88e6321_1
Initial 88e6321_1 port 1 in SGMII Mode.
Initial 88e6321_2
Reset port 3,4 Rxc delay line
 CIM powered on
 Frame Sync source assigned to Tegra-A 
 Frame Sync Enabled for all 4 Aggregators 
Initial I2C GR value for E3550 
Power on the system
Command Executed
Shell>status
Alive: 00:05:17
CPU0 Frequency: 300000000 Hz
STM0 Frequency: 100000000 Hz
CPU1 Frequency: 300000000 Hz
STM1 Frequency: 100000000 Hz
CPU2 Frequency: 300000000 Hz
STM2 Frequency: 100000000 Hz



once again, failed:

Shell>poweron
System_PowerOn: Reading PG status: Timeout Error!
X1_XA_VDD_SOC_XA_VDD_DDR2_XA_VDD_DDRQ_PG
Command Fail 
Shell>poweron x1 x2
System_PowerOn: Reading PG status: Timeout Error!
X1_XA_VDD_SOC_XA_VDD_DDR2_XA_VDD_DDRQ_PG
Command Fail 
Shell>FAN RPM reading error: voerflow flag is set

Shell>Set low threshold: 0 mv, raw value 0x0 
The board has been powered off.

 Detect KL30 voltage go below 5600 mV for 3 sample, current value is 5930 mv
System is powered off successfully because of KL30 goes down
System_PowerOn: Reading PG status: Timeout Error!
X1_XA_VDD_SOC_XA_VDD_DDR2_XA_VDD_DDRQ_PG
Set low threshold: 5600 mv, raw value 0x2fd 
Set low threshold: 0 mv, raw value 0x0 
The board has been powered off.


Dear @james.gao,
Do you see Tegra A booting(fans running)? Also, please share minicom configuration pagedetails for /dev/ttyUSB2 interface.
Also, please share complete flashing(~/.nvsdkm/) folder logs Tegra A flashing

Hi @james.gao

Please also review DRIVE AGX Developer Kit Hardware Errata and DRIVE AGX Developer Kit Mechanical & Installation Guide (PDF) and ensure that you have followed them carefully.

[quote=“SivaRamaKrishnaNV, post:5, topic:216551”]
lease share minicom configuration pagedetails for /dev/ttyUSB2 interface.

Both Fans mounted the xavier dev kit shell were running if boot the xavier B successfully, how to justifiy which fan related with Xavier A?

Please refer to the log of xavier A flashing, for the minicom, we did nothing for configuration, just as default.

NV_DRIVE_FLASH_DDPX_X1_COMP.log (7.9 KB)

Thanks you for the quick feedback and help.

Yes we know that, but not sure every time was following it as not only one engineer working on it before. so we would like to know the hardware is broken or not.

For SW9.0 relash with ubuntu 16, log as below.

NV_DRIVE_FOUNDATION_PDKINSTALLER.log (2.3 KB)
Uploading: NV_FLASH_XAVIER_PDKFLASH_A_COMP.log…
NV_FLASH_XAVIER_PDKFLASH_B_COMP.log (2.0 KB)

Dear @james.gao,

*/home/datalab/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS/drive-t186ref-foundation//tools/host/flashtools/bootburn_t19x/bootburn.sh: ./bootburn.py: /usr/bin/python: bad interpreter: No such file or directory
*

Could you double check if the file exists with execution permissions?

Hi @james.gao
Based on the Aurix voltage report Tegra A SOC voltage is down. Was this system in a car or a truck? What happened prior to the faulty behavior? When was this system purchased? Thanks.

Dear @james.gao ,
Could you provide any update?

Sorry was busy recently. the system was in a testing car, it was working well, the issue is the last few months during it working well, Xavier A was able to mount the external USB SSD disks, while Xavier B couldn’t. no other issue.

Engineer team moved the xavier back to office, and updated some code and encoding on xavier, still working well, but after that, we reboot the system then found that xavier A couldn’t boot any more, untill now.

Now acctually whole system mostly couldn’t boot at all, we purchased it at 2019.

datalab@U18:~/nvidia/nvidia_sdk526/DRIVE_OS_5.2.6_SDK_Linux_OSWithSamples_DRIVE_AGX_XAVIER/DRIVEOS/drive-t186ref-foundation/tools/host/flashtools/bootburn_t19x
$ ls bootburn.sh -l
-rwxr-xr-x 1 datalab datalab 49496 4月  24  2021 bootburn.sh