Unable to Flash R36 on Orin Nano 8GB Dev Kit Using External Storage

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

Hi,

I’m trying to flash JetPack 6.0 DP on Jetson Orin Nano 8GB Dev Kit with external storage drives but either the process gets stuck or ends with “flash failure” error. I have tried it with both NVMe SSD and Micro SD-Card.

I also tested flashing Jetson AGX Orin Dev Kit’s eMMC and it was successful. The debug log dump from SDKManager can be found as attachment. Thanks in advance.

SDKM_logs_2023-12-06_10-41-22.zip (869.2 KB)
r36_orin_nano_devkit_manual_flash.txt (311.7 KB)
r36_orin_nano_devkit_stuck.txt (71.5 KB)

2

please try to use this method to dump log from device side.

Flash issue has host side log and device side (jetson) log. So far you only shared the host side log which we cannot tell what is going on with jetson side.

3

Thanks for the quick response @WayneWWW.

I will upload device side logs today once I have my hands on a USB-TTL adapter.

4

Hi again @WayneWWW,

Please find the debug serial console debug log attached. It turns out it was just repeating usb usb2-port2: Cannot enable. Maybe the USB cable is bad? error as it is a known issue mentioned in release notes. But I just restarted my host PC and the Type C cable is the original one from Jetson AGX Orin Dev Kit’s box.
serial_console_debug.txt (71.0 KB)

5

Sorry, what is the host side log when you got stuck in this serial console state?

Actually your previous 3 host side logs show different errors.

6

Sorry I was a bit early to share serial debug log since the flashing was not over yet. Here is the synchronized debug logs from both SDKManager and serial console.
corrected_serial_console_debug.txt (86.5 KB)
SDKM_logs_2023-12-06_16-48-30.zip (1013.0 KB)

7

Hi,

Thanks for sharing. One question here. Was this host able to flash jetpack5 on same devkit here?

8

Yes, I have actually flashed back to 5.1.2 once I could not do it with JetPack 6.0 EA yesterday.

9

Could you help me try an experiment that if you connect a usb device on the type A usb ports will that make the flash process keep running?

For example, a usb keyboard or mouse shall be ok.

10

I just tried what you described with one keyboard/mouse dongle and one USB3.0 flash stick. During the flash, I did unplug-plug to see if it changes anything. It seems it looked frozen right at INFO: Flash Jetson Linux - flash: tar: Read checkpoint 490000 step, so I have unplugged and plugged both of the devices on USB ports. Meanwhile I was inspecting the serial console screen, I have noticed the changes below:

[    4.638942] usb 1-3: new full-speed USB device number 3 using tegra-xusb
[    5.082941] usb 1-2.1: new full-speed USB device number 4 using tegra-xusb
[    5.200322] input: Logitech USB Receiver as /devices/platform/bus@0/3610000.usb/usb1/1-2/1-2.1/1-2.1:1.0/0003:046D:C534.0001/input/input1
[    5.259272] hid-generic 0003:046D:C534.0001: input: USB HID v1.11 Keyboard [Logitech USB Receiver] on usb-3610000.usb-2.1/input0
[    5.263435] input: Logitech USB Receiver Mouse as /devices/platform/bus@0/3610000.usb/usb1/1-2/1-2.1/1-2.1:1.1/0003:046D:C534.0002/input/input2
[    5.263732] input: Logitech USB Receiver Consumer Control as /devices/platform/bus@0/3610000.usb/usb1/1-2/1-2.1/1-2.1:1.1/0003:046D:C534.0002/input/input3
[    5.323226] input: Logitech USB Receiver System Control as /devices/platform/bus@0/3610000.usb/usb1/1-2/1-2.1/1-2.1:1.1/0003:046D:C534.0002/input/input4
[    5.323590] hid-generic 0003:046D:C534.0002: input: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-3610000.usb-2.1/input1
[    5.403235] usb 1-2.4: new high-speed USB device number 5 using tegra-xusb
[    5.512954] usb-storage 1-2.4:1.0: USB Mass Storage device detected
[    5.514200] scsi host0: usb-storage 1-2.4:1.0
[    5.709069] tegra194-pcie 141e0000.pcie: Phy link never came up
[    6.529317] scsi 0:0:0:0: Direct-Access     Generic  Flash Disk       8.07 PQ: 0 ANSI: 4
[    6.530953] sd 0:0:0:0: [sda] 61440000 512-byte logical blocks: (31.5 GB/29.3 GiB)
[    6.531642] sd 0:0:0:0: [sda] Write Protect is off
[    6.532309] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    6.536502]  sda: sda1
[    6.539866] sd 0:0:0:0: [sda] Attached SCSI removable disk

So while these were happening in the background, SDKManager stayed frozen at INFO: Flash Jetson Linux - flash: tar: Read checkpoint 490000. Attaching the dumps again.

SDKM_logs_2023-12-06_17-37-09.zip (779.1 KB)
usb_test-serial_console_debug.txt (187.4 KB)

11

Hi,

Actually I didn’t mean you should hotplug it. I only mean you just plug the usb device since beginning and see if it passes " tar: Read checkpoint 490000 ."

Also, does it hang in checkpoint 490000 and lead to flash failure log or it was canceled by you?

1 Like
12

Here are some thoughts about this issue. If possible to try, please give it a try.
If you don’t have that device to test, then go to next item.

  1. Do you have other nvme drive to test?

  2. Do you have a usb drive to test?

  3. Do you have a sdcard to test?

If you have one of them and still get flash failure, please share me the uart log + host log again. Thanks.

Sorry in advance for the error here.

1 Like
13

We are investigating the error with initrd flash. In the mean time, these are the workaround

First workaround

First, determine what NVMe controller your NVMe SSD is on.
If your NVMe SSD is on c4 controller, use command
sudo ./flash.sh jetson-orin-nano-devkit-nvme internal

If your NVMe SSD is on c7 controller, use command
sudo UPHYLANE=c7x2 ./flash.sh jetson-orin-nano-devkit-nvme internal

Since you have tried the first flash.sh command and it gets stuck, I think you NVMe SSD is on the c7 controller so you can use that.

The downside with using this approach is that you cannot ever remove that SSD from that NVMe slot, otherwise you cannot boot the device. If you are not ok with that, run this flash command to reflash the qspi:

Second workaround
First flash the qspi:

sudo ./flash.sh -c bootloader/generic/cfg/flash_t234_qspi.xml --no-systemimg jetson-orin-nano-devkit-nvme internal

Second, follow Flashing Support — NVIDIA Jetson Linux Developer Guide 1 documentation to flash the NVMe SSD by attaching the NVMe SSD to a host PC

2 Likes
14

Hi @WayneWWW @lhoang,

Thank you for all the help. I have directly tested the workarounds mentioned by @lhoang and started with the c7 controller command. But it also stuck at the exact same step as with the c4 controller command, when I inspected the serial console logs, the device was complaining about not having anything on c7 controller. So I have moved on to the second workaround that starts with the qspi flash. As you can see from 1177th line of the successful_flash_sh_dump.txt file, this step was successful and once it was done, the device kept rebooting itself without projecting anything from the display output. Then I put into recovery mode once again, and run the sudo ./flash.sh jetson-orin-nano-devkit-nvme internal command, the flashing process was successful, didn’t even need to plug in nvme ssd to another pc and do all that manual flash. To summerize:

(1.) $ sudo ./flash.sh -c bootloader/generic/cfg/flash_t234_qspi.xml --no-systemimg jetson-orin-nano-devkit-nvme internal

(2.) $ sudo ./flash.sh jetson-orin-nano-devkit-nvme internal

Still, you can find both logs if it is going to help for your internal investigation.
successful_flash_sh_dump.txt (251.0 KB)
successful_serial_console_dump.txt (346.2 KB)

15

Glad we could help. I made a mistake when i read your log in your first post. You actually use

sudo ./flash.sh jetson-orin-nano-devkit nvme0n1p1

Which will try to flash to an sd card on a jetson orin which does not exist on your jetson. So this command will not work for your Jetson

16

Well I thought that he command for flashing to an sd-card is something like sudo ./flash.sh jetson-orin-nano-devkit mmcblk0p1.

I just have one last question. When I inspect the disk usage on the Orin, I see that its storage seems partitioned as 64GB. Do you have any idea how it might have happened? Is there any trick to expand this 64GB to 250GB as it should?

$ df -H
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   58G  8,2G   47G  15% /
tmpfs           4,0G  123k  4,0G   1% /dev/shm
tmpfs           1,6G   28M  1,6G   2% /run
tmpfs           5,3M  4,1k  5,3M   1% /run/lock
tmpfs           800M  201k  799M   1% /run/user/1000

$ sudo lsblk 
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0    16M  1 loop 
loop1          7:1    0  35,5M  1 loop /snap/snapd/20298
loop2          7:2    0  68,5M  1 loop /snap/core22/867
loop3          7:3    0     4K  1 loop /snap/bare/5
loop4          7:4    0  64,8M  1 loop /snap/cups/981
loop5          7:5    0 158,2M  1 loop /snap/chromium/2705
loop6          7:6    0  91,7M  1 loop /snap/gtk-common-themes/1535
loop7          7:7    0 475,1M  1 loop /snap/gnome-42-2204/143
zram0        252:0    0   635M  0 disk [SWAP]
zram1        252:1    0   635M  0 disk [SWAP]
zram2        252:2    0   635M  0 disk [SWAP]
zram3        252:3    0   635M  0 disk [SWAP]
zram4        252:4    0   635M  0 disk [SWAP]
zram5        252:5    0   635M  0 disk [SWAP]
nvme0n1      259:0    0 232,9G  0 disk 
├─nvme0n1p1  259:1    0 231,4G  0 part /
├─nvme0n1p2  259:2    0   128M  0 part 
├─nvme0n1p3  259:3    0   768K  0 part 
├─nvme0n1p4  259:4    0  31,6M  0 part 
├─nvme0n1p5  259:5    0   128M  0 part 
├─nvme0n1p6  259:6    0   768K  0 part 
├─nvme0n1p7  259:7    0  31,6M  0 part 
├─nvme0n1p8  259:8    0    80M  0 part 
├─nvme0n1p9  259:9    0   512K  0 part 
├─nvme0n1p10 259:10   0    64M  0 part 
├─nvme0n1p11 259:11   0    80M  0 part 
├─nvme0n1p12 259:12   0   512K  0 part 
├─nvme0n1p13 259:13   0    64M  0 part 
├─nvme0n1p14 259:14   0   400M  0 part 
└─nvme0n1p15 259:15   0 479,5M  0 part
17

Nevermind, I was able to extend the remaining area using resize2fs. Thanks a lot.

1 Like
19

mmcblk0p1 in this command is just to indicate the storage device for the APP partition.
jetson-orin-nano-devkit contains information for which device is to be flashed.

21

Can the support confirm me that the l4t_init tool don’t work with this device with the new R36.2 ???

22

Hi apa,

It seems some bugs exist in l4t inird flash and some boards can work fine while others cannot. We are still checking this issue.