[MASS FLASH] Flashing Multiple Jetson Orin NX 8GB(QSPI+NVME) Devices Always Failed

Hi, I try to flash multiple orin nx 8gb(QSPI+NVME) devices at same time, follow this document, Flashing Support — Jetson Linux Developer Guide documentation.

1, I got the BOARDID=3767 BOARDSKU=0001 FAB=300 BOARDREV=N.2 CHIP_SKU=“00:00:00:D4” RAMCODE_ID=“00:00:00:02” info, so I use command to prepare the env.

sudo BOARDID=3767 BOARDSKU=0001 FAB=300 BOARDREV=N.2 CHIP_SKU="00:00:00:D4" RAMCODE_ID="00:00:00:02" ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p "–no-systemimg -c bootloader/t186ref/cfg/flash_t234_qspi.xml" --massflash 10 --no-flash --showlogs --network usb0 jetson-orin-nano-devkit external

The result shows ok.


2, I use “sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --showlogs --network usb0 --massflash 5 --keep” to flashing the multiple device after setup the multiple devices to recovery mode.


3, Then the error hanppened. It can not starting to create gpt for emmc.

Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for device to expose ssh …RTNETLINK answers: File exists
RTNETLINK answers: File exists
Waiting for device to expose ssh …Run command: flash on fc00:1:1:0::2
SSH ready
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /mnt/internal/flash.idx
Number of lines is 76
max_index=75
writing item=59, 6:0:primary_gpt, 512, 19968, gpt_primary_6_0.bin, 16896, fixed--0, bdf464b9652e71d85739f5896c51dfadb1d0f708
Flash failure
Cleaning up…
Keeping working dir at /home/hengyongchao/NVIDIA/jetson-linux_flash/jetson_linux_35.4.1/Linux_for_Tegra/temp_initrdflash/bootloader0 and /tmp/tmp.SQ8LAwr1iD

Relate question: Flashing to Multiple Jetson Devices - #40 by 77097962

4, LOGS:
step2.txt (7.0 KB)
step1.txt (44.1 MB)
flash_3-2.4.2_1_20231024-174032.log (6.8 KB)

Hi,

Can you please try flashing them separately to find out which device is triggering this error?
We also need serial console log during flashing here.

Hi @DaveYYY

I have try to flash only one device, it has the same situlation.

I have update the orin nx 8gb uart log here.
orin-nx-8gb-uart.log (126.0 KB)

Please help me to check this issue.

Is it a DevKit or a carrier board?
Do you have other peripherals(USB, PCIe, etc.) attached to the device?

HI @DaveYYY

Carrier board.

Do we have any clues?

Oh, I mean custom carrier board.
Do you have DevKits for validation?

I have use command :

sudo ADDITIONAL_DTB_OVERLAY_OPT=“BootOrderNvme.dtbo” ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml” --showlogs --network usb0 jetson-orin-nano-devkit internal

it flash ok, but use multiple flash command, it failed. dont know why.

sudo BOARDID=3767 BOARDSKU=0001 FAB=300 BOARDREV=N.2 CHIP_SKU=“00:00:00:D4” RAMCODE_ID=“00:00:00:02” ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p “–no-systemimg -c bootloader/t186ref/cfg/flash_t234_qspi.xml” --massflash 10 --no-flash --showlogs --network usb0 jetson-orin-nano-devkit external

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --showlogs --network usb0 --massflash 5 --keep

Are you sure jetson-orin-nano-devkit is compatible with your carrier board?
Does the error happen on all devices you have, or just a few of them?

How about this?
Does taking out --keep from the flashing command make any difference?

Hi @DaveYYY

So what’s the correct command for flashing multiple orin nx 8gb devices assuming this is a offical develop board?

this not work for me.

If a device is connected to your host PC when you generate the flashing package, then these information are not needed; they will be read directly from the device. That may be why flashing directly with initrd flash without massflash works, because these hardware config on your device may differ slightly from the sample we provide.

Hi @DaveYYY

I try to analyze the mass flash failed log, Compare it with the sigle flash log. The mass flash it failed at create gpt for emmc partition.

What’s the root cause of this? left side is mass flash log, the right side is sigle flash. I test it with one orin nx 8gb device pluged in to be flashed.

LEFT SIDE:

sudo BOARDID=3767 BOARDSKU=0001 FAB=300 BOARDREV=N.2 CHIP_SKU=“00:00:00:D4” RAMCODE_ID=“00:00:00:02” ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p “–no-systemimg -c bootloader/t186ref/cfg/flash_t234_qspi.xml” --massflash 5 --no-flash --showlogs --network usb0 jetson-orin-nano-devkit external

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --showlogs --network usb0 --massflash 5

RIGHT SIDE:

sudo ADDITIONAL_DTB_OVERLAY_OPT=“BootOrderNvme.dtbo” ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml” --showlogs --network usb0 jetson-orin-nano-devkit internal

LOGS:
flash_3-2.1.1.2_0_20231025-141536.log (41.2 KB)
flash_3-2.1.2.2_0_20231026-110726.log (6.9 KB)

@DaveYYY

I want to know why there is an internal difference between the process of flash one unit individually and flash multiple devices? Why can the same device be flashed individually under different commands, but flash multiple device always fails?

What are the correct commands and procedures for mass flash orin nx? please help to fix this issues, we have 200 orin nx devices need to be programmed. It’s urgent!

Hi,

I just tried the following workflow on my Orin Nano + SD card and it is working; I think it’d be the same with Orin NX + NVMe SSD.

I’d always suggest creating the flashing image with devices connected, so you won’t mess up with those BOARDID stuff:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device mmcblk1p1 -c tools/kernel_flash/flash_l4t_t234_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --massflash 5 --no-flash --showlogs --network usb0 jetson-orin-nano-devkit external

Then flash:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --showlogs --network usb0 --massflash 1
1 Like

Hi @DaveYYY

I used the orin NX 8gb development board, and the above commands were successful for me.

I now copy the above command on our customized development board try to flash it.

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --showlogs --network usb0 --massflash 2

About our customized development board, It does not have EEPROM, and the USB burning port integrates the HUB function. After using command 2, it will wait until timeout.

What is the reason for this?

LOGS:
flash_3-2.1.2.2_0_20231026-151731.log (7.3 KB)
flash-orinnx-8gb-failed-uart.log (51.4 KB)

Glad to hear that.

Have you checked the serial console log?
You need to do this in case you haven’t:
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/HR/JetsonModuleAdaptationAndBringUp/JetsonOrinNxNanoSeries.html#eeprom-modifications

Then re-generate the massflash package.

1 Like

Hi @DaveYYY

According to my understanding.
The first step is to collect the current development board configuration and package them into a burning package.
Step 2: Copy the programming package to multiple development boards connected later.

So, the above content page contains the configuration information of the hardware. If the hardware is somewhat different (such as development board and custom board), will the burning process fail?

How can I do this:

Step 1: flash the customized development board separately and configure some software environments and software. Then package it into a flash package(restores package).
Step 2: Flash the previous installation packages to multiple connected customized boards.

Question:

  1. What the commands the above steps one and two are?
  2. What is the difference between restoring the installation package and the previous mass flash script processing?

Definitely, and that’s why you need to turn off reading EEPROM on the carrier board.

I think you need to refer to our backup/restore tool, and the README file is located in Linux_for_Tegra/tools/README_backup_restore.txt.

  1. Answered above.
  2. They just differ in how the massflash package is generated, and the step for actually flashing should be the same.
1 Like

Hi @DaveYYY

We have been modify the eeprom for a while time ago, so cvb_eeprom_read_size already set to 0x00.

cvb_eeprom_read_size = <0x0>;

Then re-generate the massflash package, and use command 2 to mass flash one device. The new issue come out, says timeout in usb write.

[ 0.4159 ] ERROR: might be timeout in USB write.

and while time , serial console dump some info:

[0008.330] C> Boot Info Table status dump :
01111111001110001111111111111111

LOGS:

flash_3-2.1.2.2_0_20231026-160434.log (5.4 KB)
flash-failed-orin-nx-8gb-serial-console-log.txt (4.2 KB)

Hi,

If you are sure it’s Orin NX 8GB, the looks like somehow the wrong dtb file is being used:

[   0.1494 ] Added binary blob_tegra234-p3767-0000-p3768-0000-a0.dtb of size 343953

p3767-0000 means Orin NX 16GB, and it would be p3767-0001 for Orin NX 8GB.
Did you generate the flashing package with devices connected, so it reads the wright hardware info?

1 Like

Hi @DaveYYY

So I need to connected the correct board(ORIN NX 8GB), do the follow commands?

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device mmcblk1p1 -c tools/kernel_flash/flash_l4t_t234_external.xml -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml” --massflash 5 --no-flash --showlogs --network usb0 jetson-orin-nano-devkit external

And then flash multiple devices

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --showlogs --network usb0 --massflash 5

Am I riight?