Difficulty Flashing - AGX Xavier

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

This NVIDIA AGX Xavier board is pretty finnicky! The toolchain for the SDK appears to be weak when using nvme as the main “external” storage. Let me expalin a bit further …

Through the GUI sdkmanager there appears to be a call to l4t_initrd_flash.sh which continues to specify the mmc default, instead of the drop-down specified nvme external device.

Once rebooted to the flashing step here it will hang using sdk manager 1.9.1.10844 on Ubuntu 20.04.

It is possible to ssh to the Xavier over usb with this :

ssh root@fc00:1:1:0::2
password is “root”.

You will have to accept the ssh key for the first time you login to the machine – this may also be holding up the ssh command to flash to the device (the terminal for sdkmanager is unclear and does not report the ssh key-accept message)

Once inside, you can get the device online using ifconfig, assuming you are connected over ethernet and the local LAN has DHCP :

ifconfig eth0 up
dhclient eth0
ifconfig -a

Then you’re online, with the IP shown.

But, it is NOT clear how to progress and flash the generated image to the nvme drive.

Now that the Xavier is here - what command must I run to properly flash it ?

I’ve tried this:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_external.xml --erase-all --network enp0s3:10.0.2.15/24:192.168.2.57 --showlogs jetson-xavier nvme0n1p1

local LAN is behind a NAT which is the reason for the 10.x address, and 192.168.2.x is what the NAT connects to. (should just work!)

After building the image, I get a :
“Finish generating flash package.
No devices to flash”

How can it not find the device to flash ?

3

Hi,

Sorry for the problem here. Actually I am a little not sure about your situation.

The flash process shall all depends on usb port. The LAN shall not be used.

Do you have any log to tell what is exactly happened when you flashed nvme drive?

4

and actually sdkmanager shall handle all the flash work there. You don’t need to manually run the initrd flash by yourself.

5

Yes, things are not flashing properly. Now it appears that the flash tool is refusing to flash. Unsure here …


11:47:42 DEBUG: Flash Jetson Linux - flash: using adapter to install NV_L4T_FLASH_JETSON_LINUX_COMP@JETSON_AGX_XAVIER_TARGETS to /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS
11:47:42 INFO: Flash Jetson Linux - flash: [host] [ Disk Avail:126.28 GB ]
11:47:42 INFO: Flash Jetson Linux - flash: Installing component 'Flash Jetson Linux' (NV_L4T_FLASH_JETSON_LINUX_COMP)
11:47:42 INFO: Flash Jetson Linux - flash: change working directory to /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra
11:47:42 INFO: Flash Jetson Linux - flash: [ Component Install Started ]
11:47:42 INFO: Flash Jetson Linux - flash: current working directory is /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra
11:47:42 INFO: Flash Jetson Linux - flash: exec_command [host]:
11:47:42 INFO: Flash Jetson Linux - flash: **********************
11:47:42 INFO: Flash Jetson Linux - flash: #!/bin/bash
11:47:42 INFO: Flash Jetson Linux - flash: set -e
11:47:42 INFO: Flash Jetson Linux - flash: set -e
11:47:42 INFO: Flash Jetson Linux - flash: unset LC_ALL
11:47:42 INFO: Flash Jetson Linux - flash: echo sudo ./nvsdkmanager_flash.sh --storage nvme0n1p1
11:47:42 INFO: Flash Jetson Linux - flash: sudo ./nvsdkmanager_flash.sh --storage nvme0n1p1
11:47:42 INFO: Flash Jetson Linux - flash: sleep 20
11:47:42 INFO: Flash Jetson Linux - flash: if [[ '--storage nvme0n1p1' == *'--nv-auto-config'* ]]; then
11:47:42 INFO: Flash Jetson Linux - flash: sleep 40
11:47:42 INFO: Flash Jetson Linux - flash: fi
11:47:42 INFO: Flash Jetson Linux - flash: **********************
11:47:42 INFO: Flash Jetson Linux - flash: exec_command: /tmp/tmp_NV_L4T_FLASH_JETSON_LINUX_COMP.sh
11:47:42 INFO: Flash Jetson Linux - flash: sudo ./nvsdkmanager_flash.sh --storage nvme0n1p1
11:47:42 INFO: Flash Jetson Linux - flash: user entered nvme0n1p1
11:47:42 INFO: Flash Jetson Linux - flash: *** Checking ONLINE mode ... OK.
11:47:42 INFO: Flash Jetson Linux - flash: *** Checking target board connection ... 1 connections found.
11:47:42 INFO: Flash Jetson Linux - flash: *** Reading ECID ... FUSELEVEL=fuselevel_production hwchipid=0x19 bootauth=NS
11:47:42 INFO: Flash Jetson Linux - flash: *** Reading EEPROM ... --- Copying tegra194-mb1-soft-fuses-l4t.cfg succeeded.
11:47:42 INFO: Flash Jetson Linux - flash: "/home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/bootloader/tegraflash.py" --chip 0x19 --applet "/home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/bootloader/mb1_t194_prod.bin" --soft_fuses tegra194-mb1-soft-fuses-l4t.cfg --bins "mb2_applet nvtboot_applet_t194.bin" --skipuid --cmd "dump eeprom boardinfo cvm.bin; dump eeprom baseinfo bbd.bin; reboot recovery"
11:47:42 INFO: Flash Jetson Linux - flash: Welcome to Tegra Flash
11:47:42 INFO: Flash Jetson Linux - flash: version 1.0.0
11:47:42 INFO: Flash Jetson Linux - flash: Type ? or help for help and q or quit to exit
11:47:42 INFO: Flash Jetson Linux - flash: Use ! to execute system commands
11:47:42 INFO: Flash Jetson Linux - flash:
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0187 ] Generating RCM messages
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0216 ] tegrahost_v2 --chip 0x19 0 --magicid MB1B --appendsigheader /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/bootloader/mb1_t194_prod.bin zerosbk
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0223 ] Header already present for /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/bootloader/mb1_t194_prod.bin
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0402 ] tegrasign_v3.py --key None --getmode mode.txt
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0405 ] Assuming zero filled SBK key
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0429 ] tegrasign_v3.py --file /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/bootloader/mb1_t194_prod_sigheader.bin --key None --length 1136 --offset 2960 --pubkeyhash pub_key.key
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0431 ] Assuming zero filled SBK key
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0523 ] Warning: pub_key.key is not found
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0539 ] tegrahost_v2 --chip 0x19 0 --updatesigheader /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/bootloader/mb1_t194_prod_sigheader.bin /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/bootloader/mb1_t194_prod_sigheader.hash zerosbk
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0870 ] tegrabct_v2 --chip 0x19 0 --sfuse tegra194-mb1-soft-fuses-l4t.cfg.pdf sfuse.bin
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0950 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x19 0 --sfuses sfuse.bin --download rcm /home/beadon/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_XAVIER_TARGETS/Linux_for_Tegra/bootloader/mb1_t194_prod_sigheader.bin 0 0
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.0959 ] RCM 0 is saved as rcm_0.rcm
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.1067 ] RCM 1 is saved as rcm_1.rcm
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.1067 ] RCM 2 is saved as rcm_2.rcm
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.1067 ] List of rcm files are saved in rcm_list.xml
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.1067 ]
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.1067 ] Signing RCM messages
11:47:42 INFO: Flash Jetson Linux - flash: [ 0.1253 ] tegrasign_v3.py --key None --list rcm_list.xml --getmontgomeryvalues montgomery.bin --pubkeyhash pub_key.key

I’ve just run the suggested option of :

sudo bash ./flashcmd.txt

$ sudo bash ./flashcmd.txt
[sudo] password for beadon: 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 

 Entering RCM boot

[   0.0000 ] rcm boot with presigned binaries
[   0.0000 ] Boot Rom communication
[   0.0022 ] tegrarcm_v2 --chip 0x19 0 --rcm rcm_1_encrypt.rcm --rcm rcm_2_encrypt.rcm
[   0.0028 ] BR_CID: 0x880219116471b504200000000a038080
[   0.0324 ] Boot Rom communication completed
[   2.1073 ] tegrarcm_v2 --isapplet
[   2.1095 ] Applet version 01.00.0000
[   2.1865 ] Sending BCTs
[   2.1886 ] tegrarcm_v2 --download bct_bootrom br_bct_BR.bct --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt --download bct_mem mem_rcm_sigheader.bct.encrypt
[   2.1893 ] Applet version 01.00.0000
[   2.2446 ] Sending bct_bootrom
[   2.2447 ] [................................................] 100%
[   2.2470 ] Sending bct_mb1
[   2.2524 ] [................................................] 100%
[   2.2569 ] Sending bct_mem
[   2.3032 ] [................................................] 100%
[   2.3500 ] Generating blob
[   2.3552 ] tegrahost_v2 --chip 0x19 --generateblob blob.xml blob.bin
[   2.3563 ] number of images in blob are 13
[   2.3570 ] blobsize is 49089503
[   2.3571 ] Added binary blob_nvtboot_recovery_cpu_t194_sigheader.bin.encrypt of size 233040
[   2.3731 ] Added binary blob_nvtboot_recovery_t194_sigheader.bin.encrypt of size 206016
[   2.3733 ] Added binary blob_preboot_c10_prod_cr_sigheader.bin.encrypt of size 24016
[   2.3735 ] Added binary blob_mce_c10_prod_cr_sigheader.bin.encrypt of size 145184
[   2.3736 ] Added binary blob_mts_c10_prod_cr_sigheader.bin.encrypt of size 3430416
[   2.3757 ] Added binary blob_bpmp-2_t194_sigheader.bin.encrypt of size 1007392
[   2.3765 ] Added binary blob_tegra194-a02-bpmp-p2888-a04_lz4_sigheader.dtb.encrypt of size 90608
[   2.3779 ] Added binary blob_spe_t194_sigheader.bin.encrypt of size 95232
[   2.3789 ] Added binary blob_tos-optee_t194_sigheader.img.encrypt of size 914992
[   2.3800 ] Added binary blob_eks_t194_sigheader.img.encrypt of size 5136
[   2.3808 ] Added binary blob_boot.img of size 42065920
[   2.3990 ] Added binary blob_kernel_tegra194-p2888-0001-p2822-0000.dtb of size 391063
[   2.4029 ] Added binary blob_tegra194-p2888-0001-p2822-0000_sigheader.dtb.encrypt of size 480272
[   2.4283 ] Sending bootloader and pre-requisite binaries
[   2.4309 ] tegrarcm_v2 --download blob blob.bin
[   2.4317 ] Applet version 01.00.0000
[   2.5097 ] Sending blob
[   2.5098 ] [................................................] 100%
[  10.4128 ] tegrarcm_v2 --boot rcm
[  10.4150 ] Applet version 01.00.0000
[  10.5074 ] RCM-boot started

The Xavier AGX now has a console (wow, progress!), but not the full OS booting from the NVME drive. I am re-running the sdkmanager tool to see if it will properly flash the OS … if this fails, what do I need to do to properly flash the OS when the AGX is in this state ?

6

Hi,

Could you dump the full log instead of these partial logs? There is a e button of “EXPORT LOGS” on sdkmanager. Please attach the whole tarball here.

Also, could we try a simple case first? Could we check if sdkmanager can flash the emmc on your board? I know you want to flash nvme, but your log seems indicating even flashing the emmc part is failed.

Just in case you don’t know. Even “flashing nvme” will flash emmc in the beginning. As the bootloader needs to be installed on the board first so that it can flash your nvme.

And actually I am not sure who taught you or what guidance you are following… For example, it is really rare to run flashcmd.txt directly. Mostly, we suggest using flash.sh.

7

I see, no , I was not aware that emmc was still required.

Just checking my information here - the EMMC is the SD CARD, correct ?

Working on the default EMMC flash now…

8

Ok, it seems that when flashing EMMC, it appears to hang(?) after flashing. The terminal shows 100% flashed, but the meter in the sdkmanager continues to progress to 98 and 99% (presumably because it;s based on some timer?), and then the sdkmanager complains the process is not responding.

Attached are the compressed logs. Thank you in advance.

SDKM_logs_2023-02-20_12-59-37.zip (232.1 KB)

9

the EMMC is the SD CARD,

No, they are different items. The emmc is inside the jetson AGX Xavier module.

10

Hi,

I checked your log and it indeed hangs. To check why it gets hang, I may need you to setup the serial console log for me.

It is simple for AGX Xavier devkit. Just prepare micro usb cable which you are sure it has data line. Connect it to the micro usb port on jetson AGX.

Details on this website

The process here is you open the console log, and start the flash process. The console shall spew info as log, then share me the log.

Thanks.

1 Like
11

Ok, here are the log messages.

ended up using :
screen /dev/ttyUSB3 115200 cs8 ixoff

logfile.out (9.9 KB)

[0103.791] I> Boot_device: SDMMC_BOOT instance: 3
[0103.791] I> sdmmc-3 params source = boot args
[0105.181] E> OCR failed, error = f0f0a06
[0105.181] E> Failed to open sdmmc-3, err = f0f0a06
[0105.181] E> Failed to initialize boot device
[0105.182] E> Top caller module: SDMMC, error module: SDMMC, reason: 0x06, aux_info: 0x0a
[0105.182] I> TBoot-CPU Recovery hang

so, it can’t open the EMMC ?

12

Hi,

I see something other suspicious log about this device so need to ask this question first:

→ Is this board really a NV developer kit or you cannot tell? It seems a custom board.

BTW, a custom board’s definition means it is not made by NV. It could be some some other vendors.

And could you also try to use latest jetpack 4 version in your sdkmanager (require to use ubuntu18.04) and see if that one can flash your obard

13

I believe it is a complete NV unit. Here are the photos

PXL_20230227_184002142
PXL_20230227_1840021421920×1080 86.1 KB

PXL_20230227_184010077
PXL_20230227_1840100771920×1080 180 KB

PXL_20230227_184017477
PXL_20230227_1840174771920×1080 199 KB

14

Is your host PC actually a VM? If not, which exact release of Ubuntu is it, e.g., 20.04 or 18.04?

15

Host PC is a VM , VitualBox 6…1.38_Ubuntu r153438 , with the VirtualBox Guest additions added to the Ubuntu 20.04 LTS OS. The USB interface via Virualbox is USB 2.0 (OHCI + EHCI) controller. Device filters allow the VM to pickup the USB device automatically when it reboot, gets disconnected, etc.

The USB serial port is connected to the host hardware, with USB Micro, the FTDI Quad serial interface is seen perfectly every time, I am using screen to connect to it. This provides a bit of separation and flexibility.

I am attempting an 18.04 VM-based installation now, flashing with the latest 4.x JetPack, with similar Guest additions in the VM. I don’t have other x86 hardware available for this task.

I had been trying to install the Jetpack 5.x version, so this lower version should produce different results.

What seems weird about the log files I’ve posted ?

16

Attached is the attempt with Ubuntu 18.04, and the jetpack 4.6.3

ubuntu-18.04–jetpack-4.6.3.serialport.out (11.9 KB)

^ Correction, this was using XHCI , USB 3.0. I am using USB 2.0 now and getting slightly different results.

Any hints ?

17

Notably, I am not seeing this with the USB 2.0 interface:

��
[0042.201] W> RATCHET: MB1 binary ratchet value 4 is too large than ratchet level 2 from HW fuses.
[0042.209] I> MB1 (prd-version: 1.5.1.9-t194-41334769-73a9b7ef)
[0042.215] I> Boot-mode: Platform RCM
[0042.218] I> Chip revision : A02P
[0042.221] I> Bootrom patch version : 15 (correctly patched)
[0042.227] I> ATE fuse revision : 0x200
[0042.230] I> Ram repair fuse : 0x0
[0042.233] I> Ram Code : 0x2
[0042.236] I> rst_source : 0x0
[0042.238] I> rst_level : 0x0
[0042.242] I> USB configuration success
[0044.568] I> mb2 image downloaded
[0044.702] I> Recovery boot mode 0
[0044.719] I> Boot-device: eMMC
[0044.724] I> UPHY full init done
[0044.730] I> MB1 done

[0044.735] W> Profiler not initialized
[0044.739] I> Welcome to MB2(TBoot-BPMP) Applet (version: 00.00.2018.32-mobile-d95d9a83)
[0044.747] W> Profiler not initialized
[0044.750] I> DMA Heap @ [0x40020000 - 0x40065800]
[0044.755] I> Default Heap @ [0xd486400 - 0xd48a400]
[0044.760] W> Profiler not initialized
[0044.763] W> Profiler not initialized
[0044.767] W> Profiler not initialized
[0045.771] E> OCR failed, error = f0f0a06
[0045.775] I> SDMMC is not present.
[0045.778] I> No supported QSPI flash found
[0045.782] E> QSPI Flash: Insufficient flash size (0 MB)
[0045.787] I> QSPI Flash is not present.
[0045.838] E> Link startup dme_set failed
[0045.842] E> UFS initialization failed
[0045.846] I> UFS is not present
[0045.849] W> Profiler not initialized
[0045.852] W> Profiler not initialized
[0045.856] W> Profiler not initialized
[0045.859] W> Profiler not initialized
[0045.863] I> Entering 3p server
[0045.866] I> USB configuration success
[0046.285] I> Populate chip info
[0046.513] I> Populate eeprom info
[0046.544] I> Populate eeprom info for module cvm

18

Do you have the host side log to share as well?

19

Normally I would be skeptical about a VM, but if it is correctly done it should work.

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.