Orin NX 8 initrd massflash errors

Hi, Nvidia!

We use many Orin NX 8 and want to flash they in custom carrierboard with NVMe thrue usb2.0 interfase.

With flashing one by initrd - no problems:

$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 \
  -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" \
  --showlogs --network usb0 jetson-orin-nano-devkit internal

With massflash have a problem:

# BOARDID=3767 FAB=300 BOARDSKU=0001 CHIP_SKU="00:00:00:xx"  ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --network u --massflash 5 jetson-orin-nano-devkit nvme0n1p1

# lsusb
Bus 001 Device 032: ID 0955:7423 NVIDIA Corp. APX

# sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --network usb0 --massflash 1 --showlogs --reuse
/home/firmware/orin/Linux_for_Tegra/tools/kernel_flash/l4t_initrd_flash_internal.sh --reuse --network usb0 --usb-instance 1-2.1.1 --device-instance 0 --flash-only --network u jetson-orin-nano-devkit nvme0n1p1
Invalid target ip
Cleaning up...

Also we tried to use --network usb1, usb2 instead of usb0 - without success.
We read Flashing Support — NVIDIA Jetson Linux Developer Guide 1 documentation and cant find any info about error “Invalid target ip”.

What mean error “Invalid target ip” ?

I found my error - when creating image I used option with error “–network u” instead “–network usb0”.

But when flashing image I have another problem after start massflash:

root@srvts1:/home/firmware/orin/Linux_for_Tegra# sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --network usb0 --massflash 1 --showlogs
/home/firmware/orin/Linux_for_Tegra/tools/kernel_flash/l4t_initrd_flash_internal.sh --network usb0 --usb-instance 1-2.1.1 --device-instance 0 --flash-only --network usb0 jetson-orin-nano-devkit nvme0n1p1
# Entry added by NVIDIA initrd flash tool
/home/firmware/orin/Linux_for_Tegra/tools/kernel_flash/tmp 127.0.0.1(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
Export list for localhost:
/home/firmware/orin/Linux_for_Tegra/tools/kernel_flash/tmp 127.0.0.1

***************************************
*                                     *
*  Step 3: Start the flashing process *
*                                     *
***************************************
Waiting for target to boot-up...
Timeout
Device failed to boot to the initrd flash kernel. Please retrive the serial log during flashing to debug further.
Cleaning up...

At this time device is in FRM mode"

root@srvts1:/home/firmware/orin/Linux_for_Tegra# lsusb
Bus 001 Device 006: ID 0955:7423 NVIDIA Corp. APX

In original devkit carrierboard same problem.

Why initrd cant find device in FRM and waitin “Waiting for target to boot-up…” ?

Now I am try online image command for the external NVMe:

# sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --showlogs --no-flash --external-device nvme0n1p1 -c ./tools/kernel_flash/flash_l4t_t234_nvme_rootfs_ab.xml --external-only --append --network usb0 jetson-orin-nano-devkit external

Then flash image with another error:

# sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --network usb0 --massflash 1 --showlogs --keep

Waiting for device to expose ssh ......Waiting for device to expose ssh ...Run command: flash on fc00:1:1:0::2
SSH ready
blockdev: cannot open /dev/mmcblk0boot0: No such file or directory
[ 0]: l4t_flash_from_kernel: Serial Number: 1421224247652
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /mnt/internal/flash.idx
Number of lines is 79
max_index=78
writing item=62, 6:0:primary_gpt, 512, 19968, gpt_primary_6_0.bin, 16896, fixed-<reserved>-0, 2b8885a93986dae4574894c82a1c77b4318f0a17
Error: Could not stat device /dev/mmcblk0 - No such file or directory.
Flash failure
Either the device cannot mount the NFS server on the host or a flash command has failed. Debug log saved to /tmp/tmp.KiGBV8pn30. You can access the target's terminal through "sshpass -p root ssh root@fc00:1:1:0::2"
Cleaning up...
Keeping working dir at /home/firmware/orin/Linux_for_Tegra/temp_initrdflash/bootloader0 and /tmp/tmp.rfzNfgr53s

Why script try to use /dev/mmcblk0 with error instaed of NVMe ?

Hi aanistratenko,

What’s the Jetpack version in use?

Could you connect the board in force recovery state w/o specifying its BOARD ID/SKU/FAB and generate mfi package from host?

Could you disable the firewall on your host to check if it could help for this error?

$ sudo ufw disable

Hello

What’s the Jetpack version in use?

Use R36.3.0

Could you connect the board in force recovery state w/o specifying its BOARD ID/SKU/FAB and generate mfi package from host?

Yes, I tried with same problem.
Orin NX is in FRM:

root@srvts1:/home/firmware/orin/Linux_for_Tegra# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 019: ID 0955:7423 NVIDIA Corp. APX

Creating image:

./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --network usb0 --massflash 5 jetson-orin-nano-devkit nvme0n1p1

Massflash package is generated at /home/firmware/orin/Linux_for_Tegra/mfi_jetson-orin-nano-devkit.tar.gz
Success
Cleaning up...
Finish generating flash package.
Error: Could not stat device /dev/mmcblk0 - No such file or directory.
Flash failure
Either the device cannot mount the NFS server on the host or a flash command has failed. Debug log saved to /tmp/tmp.dE563AwgFf. You can access the target's terminal through "sshpass -p root ssh root@fc00:1:1:0::2" 
Cleaning up...

full log flash_1-3_0_20241121-160505.log (7.1 KB)

Could you disable the firewall on your host to check if it could help for this error?

Yes, it is disabled by default
tmp.dE563AwgFf.txt (3.1 KB)

Please also share the full logs when you are generating the mfi package.

Do you have the devkit to reproduce this issue to specify if the issue is specific to your custom carrier board?

Please also share the full logs when you are generating the mfi package.

Ok, please advice where can I find its after generating image ?

Do you have the devkit to reproduce this issue to specify if the issue is specific to your custom carrier board?

Yes, last experiment I did with devkit carrier board with NVMEe for clear testing and had the same problem.

We would suggest you perform the test again and save the required message on your host.

Please share the result of cat /etc/nv_boot_control.conf on the devkit which you can reproduce the issue.

Ok, log attached.
[ssh root@10.32.1.25] (2024-11-26_143952) srv-vts-1-key.log (19.9 MB)

Please share the result of cat /etc/nv_boot_control.conf on the devkit which you can reproduce the issue.

File from host PC attached “/home/firmware/orin/Linux_for_Tegra/rootfs/etc/nv_boot_control.conf”
nv_boot_control.conf.txt (145 Bytes)

Would it work if you just run initrd flash to flash NVMe SSD on your board?

Yes, with initrd one Orin with same SSD no problems.

$ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 \
  -c tools/kernel_flash/flash_l4t_t234_nvme.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" \
  --showlogs --network usb0 jetson-orin-nano-devkit internal

Could you help to compare these 2 logs (failed with massflash, passed wiht initrd flash) for any difference?

massflash should have similar logic as initrd flash to flash the board.
You can also add the following line in l4t_flash_from_kernel.sh before running massflash to capture more logs.

# Usage: ./l4t_flash_from_kernel.sh
 # This script flashes the target from the Network File Systems on the target or
 # from the host using the images inside the flash package generated by
 # l4t_create_flash_image_in_nfs
 
set -e
+ set -x

Do you have other SoM to reproduce the similar issue to clarify if the issue is specific to current SoM?

Hello,
thanks for recommendation!

I added set -e and set -x to l4t_flash_from_kernel.sh

# Usage: ./l4t_flash_from_kernel.sh
# This script flashes the target from the Network File Systems on the target or
# from the host using the images inside the flash package generated by
# l4t_create_flash_image_in_nfs
set -e
set -x
set -eo pipefail

And try to flash one more time with

root@srvts1:~# sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only --network usb0 --massflash 1 --showlogs

with same error

writing item=62, 6:0:primary_gpt, 512, 19968, gpt_primary_6_0.bin, 16896, fixed-<reserved>-0, 14f63b24870e4803256b82811258ce096a548d0f
+ has_gpt_supported_dev=1
+ disk=/dev/mmcblk0
+ '[' 6 = 9 ']'
+ '[' -z 0 ']'
+ entry_id=0
+ '[' primary_gpt = primary_gpt ']'
+ echo -n 'writing item=62, 6:0:primary_gpt, 512, 19968, gpt_primary_6_0.bin, 16896, fixed-<reserved>-0, 14f63b24870e4803256b82811258ce096a548d0f
'
+ flock -w 60 /var/lock/nvidiainitrdflash parted -s /dev/mmcblk0 mklabel gpt
Error: Could not stat device /dev/mmcblk0 - No such file or directory.
+ cleanup

I compared massfalsh log with initdt log part:

+ has_gpt_supported_dev=1
+ disk=/dev/mmcblk0
+ '[' 9 = 9 ']'
++ get_disk_name nvme0n1p1
++ local ext_dev=nvme0n1p1
++ local disk=
++ [[ nvme0n1p1 = sd* ]]
++ disk=nvme0n1
++ echo nvme0n1
+ disk=/dev/nvme0n1
+ '[' -z 0 ']'
+ entry_id=0
+ '[' master_boot_record = primary_gpt ']'
+ '[' master_boot_record = secondary_gpt ']'

Full logs attached
[ssh root@10.32.1.25] (2024-12-09_123258) srv-vts-1-key_initrd success.log (801.9 KB)
[ssh root@10.32.1.25] (2024-12-09_122740) srv-vts-1-key_massflash fail with log.log (98.5 KB)