Jetson Orin Nano Developmen Kit - NVMe flash fails in every possible way

[m.j.kurz]

Hello community,

so the new Jetson Orin Nano DevKits have arrived and we were trying to flash it in multiple ways - namely:

• sdkmanager cli

sdkmanager --cli install --logintype devzone --product Jetson --version 5.1.1 --targetos Linux --target JETSON_ORIN_NANO_TARGETS --flash all

…which either:

** fails already to start, when only choosing what we need (Jetson Linux, Flash) with a weird error like: something something not detected and: TypeError: Cannot read properties of undefined (reading 'show')

** if one chooses more components (for example stuff from the Runtime) the download and installation starts, but then fails completely with:

info: Found /usr/bin/qemu-aarch64-static                             │
│   Exit                                                              ││info: Installing QEMU binary in rootfs                               │
│                                                                     ││error: mknod: /home/supergrobi/nvidia/nvidia_sdk/JetPack_5.1.1_Linux │
│                                                                     ││_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/rootfs/dev/random          │
│                                                                     ││error: : File exists                                                 │
│                                                                     ││info: [ Package Install Finished with Error ]                        │
│                                                                     ││info: [host] [ 14.84 MB released. Disk Avail: 21.54 GB ]             │
│                                                                     ││info: [ NV_L4T_FILE_SYSTEM_AND_OS_COMP Install took 4m49s ]          │
│                                                                     ││info:                                                                │
│                                                                     ││error: command terminated with error                                 │
│                                                                     ││info: Installation failed.                                           │
│                                                                     ││info: Depends on failed component  

** if one deletes everything and starts from scratch, the download and installations starts, but then gets stuck and loops at (for 12hours - if you want to…):

tegrarcm_v2 --chip 0x23 0 --ismb2applet
tegrarcm_v2 --chip 0x23 0 --ismb2

• Setting up the manual way according to: Quick Start — Jetson Linux Developer Guide documentation
  and then (from the Linux_for_Tegra-folder):
  ** sudo ./flash jetson-orin-nano-devkit nvme0n1p
  which gets stuck at:
  flash-sh-jetson-orin-nano-devkit-nvme0n1p1.txt (72.0 KB)

** sudo ./tools/kernel_flash/l4t_initrd_flash.sh jetson-orin-nano-devkit nvme0n1p
fails with:
tools-kernel-flash.txt (157.1 KB)

** sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal
which fails (depending on which colleague is working on it) either with:

ubuntu@ubuntu:~/nvidia/nvidia_sdk/JetPack_5.1.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal
/home/ubuntu/nvidia/nvidia_sdk/JetPack_5.1.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/tools/kernel_flash/l4t_initrd_flash_internal.sh --no-flash --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p -c bootloader/t186ref/cfg/flash_t234_qspi.xml --showlogs --network usb0 jetson-orin-nano-devkit internal
exportfs: /home/ubuntu/nvidia/nvidia_sdk/JetPack_5.1.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra/rootfs does not support NFS export
Cleaning up...

…or with:
tools-kernel-flash-qspi-orin.txt (196.8 KB)

…

Could someone please shed some competence on our issues? As you might have assumed: we are trying to install the software on the external NVMe. Force Recovery mode should be fine (we place the jumper on the FR-pins, and remove it after we have supplied the board and checked it with lsusb.

Thanks, have a great day and lovely regards.

[DaveYYY]

Hi,

this user had nearly the same issue as you did:

Failed to flash Orin Nano Developer Kit

We’re trying to flash the Orin Nano Developer Kit using this command: sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device mmcblk1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal *reference :https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/IN/QuickStart.html#to-flash-the-jetson-developer-kit-operating-software We see this message Linux_for_Tegra/rootfs/li…

Can you run sudo ./apply_binaries.sh again, and check if those missing files are back?
If it still does not work, download and extract the BSP package again.

[m.j.kurz]

Hello Dave,

and thanks for your answer.

So I downloaded:
Jetson_Linux_R35.3.1_aarch64.tbz2
and
Tegra_Linux_Sample-Root-Filesystem_R35.3.1_aarch64.tbz2
…unpacked it, according to the manual and executed the scripts:

• sudo ./apply_binaries.sh
  apply-binaries-sh.txt (724 Bytes)
• sudo ./tools/l4t_flash_prerequisites.sh
  l4t-flash-preq.txt (1.8 KB)

The result though stays the same:

• sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device mmcblk1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal
  sudo-tools-kernel-l4t-initrld-flash-l4t-external.txt (235.5 KB)

• cd bootloader and sudo bash ./flashcmd.txt
  bootloader-flashcmd.txt (2.9 KB)

[DaveYYY]

Hi,

is apply-binaries-sh.txt a full log of what is shown in terminal after you run sudo ./apply_binaries.sh? It should install way more packages than what your log gives. Did you delete the original folder and have a freshly downloaded-extracted BSP?

Can you show us what is inside your Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers folder?

[m.j.kurz]

Thanks for your correspondence.

The apply-binaries-sh.txt was indeed the full log.

I have downloaded both packages again from:
https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v3.1/release/jetson_linux_r35.3.1_aarch64.tbz2/

…with…

md5sum Jetson_Linux_R35.3.1_aarch64.tbz2
1d254bec244e1bdeaa4012f02e2c6057  Jetson_Linux_R35.3.1_aarch64.tbz2

and:

https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v3.1/release/tegra_linux_sample-root-filesystem_r35.3.1_aarch64.tbz2/

…with…

md5sum Tegra_Linux_Sample-Root-Filesystem_R35.3.1_aarch64.tbz2
12bd3ed977eb86f6b13e90a2000022de  Tegra_Linux_Sample-Root-Filesystem_R35.3.1_aarch64.tbz2

This time I figured out, that when first time, after unpacking the archives freshly the log mildly differs:

230517_apply-binaries_dif.txt (1.8 KB)

Following your suggestion to ls -al Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers we indeed found something unexpected - namely that:

supergrobi@OS:~/nvidia$ ls -al Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers
ls: cannot access 'Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers': No such file or directory
supergrobi@OS:~/nvidia$ ls Linux_for_Tegra/rootfs/lib/modules
ls: cannot access 'Linux_for_Tegra/rootfs/lib/modules': No such file or directory

Could this be a hint to what is going wrong in here?
Further - permissions are okay like that:

supergrobi@OS:~$ ls -al nvidia/Linux_for_Tegra/rootfs
total 16
drwxr-xr-x 1 root       root        150 May 17 08:51 .
drwxr-xr-x 1 supergrobi supergrobi 4052 Mar 19 15:23 ..
-rw-r--r-- 1 supergrobi supergrobi   62 Mar 19 15:14 README.txt
lrwxrwxrwx 1 root       root          7 Jul 31  2020 bin -> usr/bin
drwxr-xr-x 1 root       root         16 May 17 08:51 boot
drwxr-xr-x 1 root       root         26 May 17 08:51 dev
drwxr-xr-x 1 root       root       3882 Mar  1 03:21 etc
drwxr-xr-x 1 root       root          0 Apr 15  2020 home
lrwxrwxrwx 1 root       root          7 Jul 31  2020 lib -> usr/lib
drwxr-xr-x 1 root       root          0 May 19  2021 media
drwxr-xr-x 1 root       root          0 Jul 31  2020 mnt
drwxr-xr-x 1 root       root         12 May 17 08:51 opt
drwxr-xr-x 1 root       root          0 Apr 15  2020 proc
drwx------ 1 root       root         42 Sep 20  2022 root
drwxr-xr-x 1 root       root        416 Jul  7  2021 run
lrwxrwxrwx 1 root       root          8 Jul 31  2020 sbin -> usr/sbin
drwxr-xr-x 1 root       root          0 Feb  2  2021 snap
drwxr-xr-x 1 root       root          0 Jul 31  2020 srv
drwxr-xr-x 1 root       root          0 Apr 15  2020 sys
drwxrwxrwt 1 root       root          0 Mar  1 03:21 tmp
drwxr-xr-x 1 root       root         84 Mar  4  2021 usr
drwxr-xr-x 1 root       root        122 Mar  4  2021 var

[DaveYYY]

That is really weird…
Absence of the lib/modules means your system is no more than a pure Ubuntu, with no NVIDIA stuff in it.
Are you sure sudo ./apply_binaries.sh end correctly without any errors? Also try if this post helps.
If downloading and extracting the BSP package does not solve the issue, can you please try to use a different host PC?

[m.j.kurz]

Thanks David - this helped a little - but not fully yet.

sudo ./apply_binaries.sh

resultet in great success:
230517_apply-binaries_suc.txt (12.7 KB)

tough the flashing procedure…

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device mmcblk1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal

…looks different than usual - but still fails:
230517_sudo-flash-sh-jetson-orin-nano_2.txt (246.0 KB)

sudo bash ./flashcmd.txt

230517_bootl-sudo-flashcmd.txt (1.6 KB)

[DaveYYY]

Hi,

sorry for the late reply. Can you attach log dumped from UART when flashing the board?

[ss332]

@m.j.kurz What is the host system? It seems NVMe flashing is just broken out of the box for Ubuntu 20. I faced yet another failure mode flashing to NVMe.

A friend has successfully flashed with Ubuntu 18 but we need to repeat it on another machine to be sure if the OS is truly the issue.

[ss332]

Also, try this as a workaround if everything else fails

(BLOG) - Boot from NVMe without using SDKManager or external Ubuntu PC (A solution that works)

Hi All, I’ve done it again :D . This time, I’ve scripted the above steps making it even simpler, and have added scripts that will install NVIDIA Jetpack and Deepstream. ( i.e. covering all features of the SDKManager ) once the board is rebooted. You can find the scripts and instructions in my git repo below: Feel free to reachout if you have any questions, and i’ll do my best in answering. The above script only works (or tested on) NVIDIA Jetson Orin Nano, so would be interested if it works…

[ss332]

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1 -c ./tools/kernel_flash/flash_l4t_nvme.xml -S 200GiB --showlogs --erase-all jetson-orin-nano-devkit nvme0n1p1

Fails in the same manner I have seen in SDKM, citing permissions issues with RTELink. Why is this so broken on Ubuntu 20? It seems Ubuntu 18 works fine:

Jetson Orin Nano Dev Kit 8GB Unable to Flash Through SDK Manager

My Jetson Orin Nano Dev Kit (microSD card version) had the same issue from another post. I followed the steps from the documentation provided by NVIDIA. I used a host machine with Ubuntu 18.04. I tried both automatic setup and manual setup with the jumper. However, I have been running into an issue whenever I get to the “flash” part of the setup with the following message. 07:07:06 INFO: Flash Jetson Linux - flash: *** Reading EEPROM ... "/home/derek/nvidia/nvidia_sdk/JetPack_5.1.1_Linux_JETSON…

Sdkmanager flash error

Please let us know the result, thank you. How did you upgrade from 35.1 to 35.2? eMMC should be recognized because you could boot up with NVMe drive, there should be some boot contents in eMMC for boot up. NVMe is only used for rootfs.

[m.j.kurz]

Hello Dave :),
please find the serial-log attached:
230523_serial.log (30.2 KB)

Hi ss332,
sorry for the late reply from my side and thanks for your correspondence. So regarding the host systems that I have tried:

• Ubuntu 18
• Ubuntu 20
• NixOS (with Ubuntu 18 or 20 in Docker - for the Jetson Nano this setup worked perfectly fine).

[DaveYYY]

Hi,

Looks like the flashing process was interrupted as your device entered the UEFI Shell.
Make sure UEFI Shell is not in the first place in boot order, and unplug all peripherals (keyboard, mouse, USB flash drive, etc.) from your device and flash again.

[m.j.kurz]

Hello,

so I have tried the whole process now without any peripherals attached (keyboard, mouse, USB flash drive, Ethernet etc).

230601_ab_flash_jet_or_n.log (256.5 KB)

The flash seems to succeed and seems to provide an open ssh connection.
The IP is not requested from the DHCP server (that I have configured on my Notebook) - but set static from a previous flash (192.168.0.1).

ssh root@192.168.0.1

…does not respond.

Restarting the device and showing the logs over the serial monitor:
230523_serial.log (30.2 KB)
…shows me again, that the normal boot is interrupted by UEFI. As I try to manually exchange the boot order in the UEFI-menu (haahaa), I see, that the NVMe should be the first to boot, then comes IPv4, then IPv6 and so on.

What seems miraculous in that context is, that even though I flashed the device now multiple times - the boot-loader remains the UEFI with the previously set configurations.

Is there any way to flash the device even harder?

Thanks in advance and have a great day.

[DaveYYY]

@m.j.kurz

The IP is not requested from the DHCP server (that I have configured on my Notebook) - but set static from a previous flash (192.168.0.1).

I don’t quite get it with this part, and can you please elaborate it more?
You should be able to ssh into 192.168.55.1 if you connect the device to your host PC with a micro USB cable.

@m.j.kurz

…shows me again, that the normal boot is interrupted by UEFI. As I try to manually exchange the boot order in the UEFI-menu (haahaa), I see, that the NVMe should be the first to boot, then comes IPv4, then IPv6 and so on.

What seems miraculous in that context is, that even though I flashed the device now multiple times - the boot-loader remains the UEFI with the previously set configurations.

So does manually changing boot order work?

[m.j.kurz]

Thanks for asking these questions - I got it wrong before and tried to establish a connection through Ethernet-SSH.

Unfortunately it seems, that my Host-PC is not getting the CDC_USB up. One issue I figured out already was, that I didn’t have TPM2 activated on my host. After activating I still get:

usb usb2-port1: config error

…when dmesg --follow and plugging in the Jetson USB.

Regarding the UEFI shell - yes - I could change the boot-order:

The serial log though stays basically the same (not booting into NVMe - but coming up in the end with the UEFI-shell).

[DaveYYY]

@m.j.kurz

usb usb2-port1: config error

Did dmesg give you things in addition to this when the connection failed?
Have you tried with a different USB cable/port?

I’d suggest you use UART or real Ethernet to log into your device if the USB method is temporarily unavailable.

[m.j.kurz]

Did dmesg give you things in addition to this when the connection failed?
Have you tried with a different USB cable/port?

dmesg --follow-new gives the same error (and only that):
usb usb2-portX: config error
…on all USB-ports.

I’d suggest you use UART or real Ethernet to log into your device if the USB method is temporarily unavailable.

UART works fine. The main issue - that even after flashing the devices NVMe, it does not boot, but just opens the promp to press either ESC, F11 or Enter to either [go into boot-menu], [continue normal boot], [open UEFI-shell] - persists.

The IP is not requested from the DHCP server (that I have configured on my Notebook) - but set static from a previous flash (192.168.0.1).

I don’t quite get it with this part, and can you please elaborate it more?

To elaborate further on that: In one of my previous efforts I have set the IP in the UEFI-boot-menu to 192.168.0.1.
After that - I have made a series of attempts to flash the devices NVMe with the, in the previous comments mentioned methods. Further I have set up a DHCP-server on my host - so when connecting the physical Ethernet cable - the Jetson should request an IP from the DHCP.

In practice, this does not happen - and the IP remains 192.168.0.1 (its possible to ping) - and trying to connect with ssh root@192.168.0.1 fails.

[DaveYYY]

@m.j.kurz

so I have tried the whole process now without any peripherals attached (keyboard, mouse, USB flash drive, Ethernet etc).

230601_ab_flash_jet_or_n.log (256.5 KB)

The flash seems to succeed and seems to provide an open ssh connection.

Hi, sorry that I just noticed the flashing process was not complete.
It booted into initrd kernel, and rootfs was to be flashed via SSH, which should be established with the USB cable.

Did you get anything after

Waiting for device to expose ssh …RTNETLINK answers: File exists
RTNETLINK answers: File exists
Waiting for device to expose ssh …Run command: flash on fc00:1:1:0::2
SSH ready

, or you thought it had completed so you removed the cable?

[m.j.kurz]

I indeed thought it had completed - though trying it again it basically adds up:

Flash failure
Cleaning up...

…in the end of the log.

20230609_flsh_jet.log (247.2 KB)

Is the command:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device mmcblk1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal

…(in specific - based on the assumption, that eth-over-usb does not work - --network usb0) correct?

Thanks a lot for your efforts.