Orin NX initrd flashing too slow

Robotics & Edge Computing Jetson Systems Jetson Orin NX
1

Hey there,

I’m currently facing an issue regarding flashing speed of a Jetson Orin NX 16GB with the Orin Nano devkit.

I have successfully managed to clone a rootfs and I am trying to flash it in to a newer board with some custom made partitions for factory data.

The flash goes on as expected but the writting speed of 3MB/s is to slow even for USB2.0, considering a 20G APP image. Are this speeds normal? Is there any way to speedup the flashing?

Copied 109 bytes from /mnt/internal/qspi_bootblob_ver.txt to address 0x03fe0000 in flash
00:04:29.187 - Debug: Writing gpt_secondary_3_0.bin (parittion: secondary_gpt) into /dev/mtd0
00:04:29.194 - Debug: Sha1 checksum matched for /mnt/internal/gpt_secondary_3_0.bin
00:04:29.197 - Debug: Writing /mnt/internal/gpt_secondary_3_0.bin (16896 bytes) into  /dev/mtd0:67091968
650362880 bytes (650 MB, 620 MiB) copied, 207 s, 3.1 MB/sCopied 16896 bytes from /mnt/internal/gpt_secondary_3_0.bin to address 0x03ffbe00 in flash
00:04:29.230 - Info: Successfully flashed the QSPI.
3095322624 bytes (3.1 GB, 2.9 GiB) copied, 1036 s, 3.0 MB/s

My flashing command is the following: (partitions.xml it’s flash_l4t_t234_nvme.xml with an user defined partition and adjusted UDA size.)

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --external-device nvme0n1p1  -c tools/kernel_flash/partitions.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit external

Checking the USB Bus it shows 480M for NVIDIA Corp. I’ve already checked the cable and the bus.

 matias@matiasPC ~/  lsusb -tv
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    ID 1d6b:0002 Linux Foundation 2.0 root hub
    |__ Port 5: Dev 55, If 1, Class=CDC Data, Driver=rndis_host, 480M
        ID 0955:7035 NVIDIA Corp. 
    |__ Port 5: Dev 55, If 0, Class=Communications, Driver=rndis_host, 480M
        ID 0955:7035 NVIDIA Corp. 
    |__ Port 8: Dev 2, If 1, Class=Video, Driver=uvcvideo, 480M
        ID 13d3:5415 IMC Networks 
    |__ Port 8: Dev 2, If 0, Class=Video, Driver=uvcvideo, 480M
        ID 13d3:5415 IMC Networks 
    |__ Port 9: Dev 3, If 0, Class=Vendor Specific Class, Driver=, 12M
        ID 06cb:00bd Synaptics, Inc. 
    |__ Port 10: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
        ID 8087:0026 Intel Corp. 
    |__ Port 10: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M
        ID 8087:0026 Intel Corp.
2

*** Please note that this reply is generated by LLM automatically ***

The issue you’re facing with the slow flashing speed of the Jetson Orin NX 16GB with the Orin Nano devkit is not uncommon. There are several factors that could be contributing to the slow flashing speed, and I’ll outline some possible causes and solutions.

Causes of slow flashing speed:

  1. USB Bus Speed: The USB bus speed is a significant factor in determining the flashing speed. In your case, the USB bus speed is 480M, which is the maximum speed for USB 2.0. This speed is relatively slow compared to USB 3.0 or newer standards.
  2. Partition size and complexity: The size and complexity of the partitions can also impact the flashing speed. If the partitions are large or have complex configurations, it can take longer to flash the image.
  3. Image size: The size of the image being flashed is also a factor. In your case, the image size is 20G, which is relatively large.

Solutions to speed up flashing:

  1. Use a faster USB interface: If possible, try using a USB 3.0 or newer interface, which can provide faster speeds.
  2. Optimize the partition layout: Review the partition layout and optimize it for faster flashing. You can try reducing the number of partitions or simplifying the partition configuration.
  3. Use a faster flashing tool: The flash.sh tool is a wrapper script that calls the l4t_initrd_flash.sh script. You can try using the l4t_initrd_flash.sh script directly, which may provide faster flashing speeds.
  4. Use the --mass-storage-only option: The --mass-storage-only option can help speed up the flashing process by only flashing the mass storage device.
  5. Use a faster storage device: If you’re using an external storage device, try using a faster device, such as an NVMe SSD.

Comparison with other Jetson modules:

The flashing speed of the Jetson Orin NX 16GB with the Orin Nano devkit is likely to be similar to other Jetson modules with similar configurations. However, the flashing speed can vary depending on the specific module, storage device, and partition layout.

Example commands to speed up flashing:

Here are some example commands that you can try to speed up the flashing process:

  • Using the l4t_initrd_flash.sh script directly:
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --mass-storage-only --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" --showlogs --network usb0 p3509-a02-p3767-0000 internal
  • Using the --mass-storage-only option:
sudo ./flash.sh --mass-storage-only --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" --showlogs --network usb0 p3509-a02-p3767-0000 internal

Note that these commands are just examples, and you may need to modify them to suit your specific use case.

In summary, the slow flashing speed of the Jetson Orin NX 16GB with the Orin Nano devkit is likely due to a combination of factors, including the USB bus speed, partition size and complexity, and image size. By optimizing the partition layout, using a faster USB interface, and trying different flashing tools and options, you may be able to speed up the flashing process.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Hi malmarcha,

What’s the Jetpack version in use?

What’s the flashing time when you run this command to flash?
What’s the expected flashing time? Are you doing the mass production so that you are caring about the flashing time?

5

Hi Kevin,
I’m currently using JP6.2.1. This is just a developement board and we are getting acquainted with the flashing procedures using initrd.
At this current speed the flash time is around 2 hours. Using USB3.0 I would expect at the most 20m.

6

For the developer kit, I don’t think it takes 2 hours for the flashing.

Are you actually using the official developer kit and flashing it with official BSP package?
Please also try using the SDK manager to flash the devkit.

7

Flashing the Jetson Linux from scratch using SDK Manager takes around ~15m. (For a 2.2G image) so I think my times are consistent with SDK Manager.

Is there a metric from NVIDIA for how long should the default Jetson Linux flash last?

For me it seems a bit off considering that we are using USB 3.0 and the lsusb show the board connected to the USB2.0 root hub of my USB3.0 at 480M

PD: We are using the NVIDIA provided USB cable and a certified USB3.0 hub from Lenovo

8

“15 mis” is expected time to me that it needs to download and create the package before the actual flashing.

Why you said that it takes 2hrs for the flashing bofore?

9

I see that there are a lot of devices connected to this one hub. There also seems to be some video and BT/WIFI devices that could have an impact on the flashing time of the Jetson device if their are running at the same time as your flashing process.

An USB3 connection won’t give you any significant improvement. During the initrd phase, a minimal system is flashed over USB (the generated boot*.img images). This image brings up an RNDIS network interface over USB 2.0, which is then used to mount an NFS storage that is capped at around 29 MB/s read speed (this limit appears to come from the NFS kernel drivers). A single Jetson RNDIS USB 2.0 connection is enough to saturate this limit (~250 Mbit/s ≈ ~31 MB/s).

An ideal usb setup would look like this:


/:  Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 5000M
ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 07.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
|__ Port 1: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 480M
ID 0955:7323 NVIDIA Corp.
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 5000M
ID 1d6b:0003 Linux Foundation 3.0 root hub
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
1 Like
10

@KevinFFF

The “15 mins” are for the base Jetpack image that it is downloaded and flashed by default by SDKManager. This image weighs around 2.2G without any Jetson components.
We have a custom image with several files for development, it weighs around 20G therefore the flashing takes proportionally longer.

@TobidieTopfpflanze
Thank you for your insightfull answer,
Regarding the amount of devices, most of them come by default in my notebook. I tried using ionice while flashing but with no better results.
About RNDIS, are the write/read speeds symetrical? Because my writting speeds are way below, around 3MB/s.

11

I am not sure, if the log there is trustworthy. Could you please monitor your network traffic over USB by using nload (You might have to install it first with sudo apt-get install nload. It gives you a nice visualisation.

You can start nload usb0 after you see the following flashing logs:

Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for device to expose ssh ......RTNETLINK answers: File exists
RTNETLINK answers: File exists
Waiting for device to expose ssh ...Run command: flash on fc00:1:1:0::2
13

Using nload eth0 (nmcli shows eth0 as rndis_host for NVIDIA Linux for Tegra) it shows an average of ~24 to ~29MBit/s , it matches with the ~3.5MB/s displayed by the log.

14

it should be nload usb0 not eth0

15

usb0 yields no result. nmcli shows the following when entering initrd

eth0: unmanaged
        "NVIDIA Linux for Tegra"
        ethernet (rndis_host), 1A:EB:28:2F:FB:E7, hw, mtu 1500

Is this normal behaviour?

16

What’s the flashing time after you install those component?

I will suggest you refer to Workflow 3: To massflash the backup image in <Linux_for_Tegra>/tools/backup_restore/README_backup_restore.txt to create the image from one device and do mass flash to reduce the flashing time.

17

The flashing time is around 2 hours for the 20G image at ~29 MBit/s (~3Mb/s)

I will try the Workflow 3 as suggested and give you feedback.

18

Okay, I believe you don’t need 2hrs to flash the board through backup/restore script.