Error when flashing Orin NX using l4t_initrd_flash

[embedx]

Hello,

I am working over the Orin NX module + Orin Nano carrier board. I flashed the Jetpack 6 succesfully using the sdkmanager. Now I am trying to flash my own kernel and dts using l4t_initrd_flash script, but it fails.

The command that I use is:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/generic/cfg/flash_t234_qspi.xml” --showlogs --network usb0 jetson-orin-nano-devkit internal > flash_logs.log

Below you can find the last logs showed before flash failure:

Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for target to boot-up…
Waiting for device to expose ssh …RTNETLINK answers: File exists
RTNETLINK answers: File exists
Waiting for device to expose ssh …Run command: flash on fc00:1:1:0::2
SSH ready
blockdev: cannot open /dev/mmcblk3boot0: No such file or directory
[ 0]: l4t_flash_from_kernel: Starting to create gpt for emmc
Active index file is /mnt/internal/flash.idx
Number of lines is 79
max_index=78
writing item=62, 6:0:primary_gpt, 512, 19968, gpt_primary_6_0.bin, 16896, fixed--0, 1b6086b41ef3bf9e54dc6a3b5756680e610fdbfa
Error: Could not stat device /dev/mmcblk3 - No such file or directory.
Flash failure
Cleaning up…

Using Jetpack 5.1.2 I did not have this problem, the module was flashed correctly.

Any idea of what the problem might be o which way to go?

Thanks in advance.

[DaveYYY]

Hi,

Did you also remember to install kernel modules after compiling the new kernel?
Anyway, please dump the serial console log during flashing as indicated in:

Jetson Nano & NX Style - Serial Debug Console - JetsonHacks

[embedx]

Yes, I build and deployed the kernel modules.

Attached you can find the serial port logs during flashing:

serial-port-logs.txt (118.1 KB)

[DaveYYY]

Hi,

[2024-01-22_14:17:29:301]modprobe: ERROR: could not insert ‘tegra_mce’: Exec format error
[2024-01-22_14:17:29:320]9968] spi_tegra210_quad: disagrees about version of symbol module_layout

I don’t believe you are doing it right.
Please also check this post:

Can't boot orin nx after flashing custom Kernel

Hi, This is a known issue on JetPack 6 with kernel building. We are tracking it internally. This is because we need separate kernel drivers to mount NVMe/USB as rootfs during booting, and these drivers has to be present in the initrd image. However, when you build kernel/kernel modules, the initrd image is not updated, so the value of module_layout in kernel modules in the stock initrd image released by NVIDIA does not match the new kernel image you built yourself. For a temporary solution, …

[embedx]

It’s a clean kernel build. For this, I used following steps:

Prepare the workspace:

• Download BSP using sdkmanager
• Flash Orin NX first time using the sdkmanager in order to test that it works great.
• sudo apt-get install git-core
• Run ./source_sync.sh -k -t jetson_36.2 located in /home/user/nvidia/nvidia_sdk/Jetpack_6.0_DP_Linux_DP_JETSON_ORIN_NX_TARGETS/Linux_for_Tegra/source
• Move to source folder

• Install gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
• Set environment settings (CROSS_COMPILE, PATH, ARCH, LOCALVERSION)

Make a clean kernel build:

• make clean
• make defconfig
• make -j4 Image
• make -j4 modules
• sudo -E make INSTALL_MOD_PATH=~/nvidia/nvidia_sdk/Jetpack_6.0_DP_Linux_DP_JETSON_ORIN_NX_TARGETS/Linux_for_Tegra/rootfs/ modules_install
• cp kernel/kernel-jammy-src/arch/arm64/boot/Image …/kernel/
• cp kernel/kernel-jammy-src/arch/arm64/boot/dts/* …/kernel/dtb/

Flash board:

• Boot the NX to recovery mode.
• Move to ~/nvidia/nvidia_sdk/Jetpack_6.0_DP_Linux_DP_JETSON_ORIN_NX_TARGETS/Linux_for_Tegra
• Flash using: sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/generic/cfg/flash_t234_qspi.xml” --showlogs --network usb0 jetson-orin-nano-devkit internal

I will move my own compiled modules to initrd image as you suggest.

Thanks for your support!

[DaveYYY]

@embedx

Install gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz

We are moving to GCC 11.3 on JetPack 6.
https://developer.nvidia.com/downloads/embedded/l4t/r36_release_v2.0/toolchain/aarch64--glibc--stable-2022.08-1.tar.bz2
7.3 is quite old and it may cause some ABI compatibility issues on building newer kernel versions.

[embedx]

Thanks Dave!

I recompile again using GCC 11.3 and I updated the initrd as following:

Re-pack the initrd image:

• sudo cp Linux_for_Tegra/rootfs/boot/initrd ~/jp6/initrdImg/
• mkdir ~/jp6/initrdImg/unziped
• cd ~/jp6/initrdImg/unziped
• sudo gunzip -c /home/user/jp6/initrdImg/initrd | cpio -i

Copy modules from Linux_for_Tegra/rootfs/lib/modules:

• cd unziped/lib/modules
• cp -r /home/user/nvidia/nvidia_sdk/Jetpack_6.0_DP_Linux_DP_ORIN_NX_TARGETS/Linux_for_Tegra/rootfs/lib/modules/5.15.122-tegra .
• find . | cpio -H newc -o | gzip -9 -n > …/initrd

Copy initrd back to folder:

• cd …
• sudo cp initrd /home/user/nvidia/nvidia_sdk/Jetpack_6.0_DP_Linux_DP_ORIN_NX_TARGETS/Linux_for_Tegra/rootfs/boot/initrd

Flash board:

• Boot NX to recovery mode
• cd /home/user/nvidia/nvidia_sdk/Jetpack_6.0_DP_Linux_DP_ORIN_NX_TARGETS/Linux_for_Tegra
• sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/generic/cfg/flash_t234_qspi.xml” --showlogs --network usb0 jetson-orin-nano-devkit internal

But it fails again. Last dmesg logs are:

Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Waiting for target to boot-up...
Timeout
Cleaning up...

And serial port debug logs from Orin NX board:

serial-port-debug-02.txt (101.6 KB)

[DaveYYY]

OK, so now you are not getting error loading those drivers/modules.
That’s good.

Do you have any USB 2.0 devices connected to the board while flashing? Anything like a mouse or a keyboard. There’s currently a bug in 36.2 that USB devices are not enumerated correctly, which affects the flashing cable.

[embedx]

No, I don’t have any device connected to the board, only the flashing cable.

[DaveYYY]

Then please make sure there is one.
Also, are you sure it’s the complete serial console log?
I don’t think it will timeout after only 3 seconds into the kernel.

[embedx]

It doesn’t work either with a USB2.0 keyboard connected.

Yes, it is the complete serial console log. Attached you can find the last one using a keyboard connected to one USB port.

serial-port-03.txt (102.5 KB)

I will review everything anyway.

Thanks!!

[DaveYYY]

You can also consider this if it keeps failing:
https://docs.nvidia.com/jetson/archives/r36.2/DeveloperGuide/SD/FlashingSupport.html#manually-setting-up-a-flash-drive-for-booting

[embedx]

Orin NX is based on NVMe storage and now I don’t have any adapter to connect it directly to the host. I will get one to test.

[embedx]

Anyway, I suppose that it will be solved in the next Jetpack version. Is the next version scheduled?

[DaveYYY]

It’s scheduled to be released in March.

[embedx]

Thank you so much Dave!