JP 6 Flash Issue NVME

Hello,

I have been trying to flash JP6 with RT patch on NVME, it fails to boot. I am attaching log file generated. Flashing to internal memory works good.
AGX ORIN 64GB Dev kit

Commands used:

sudo tools/l4t_flash_prerequisites.sh
sudo ./apply_binaries.sh
sudo ./tools/kernel_flash/l4t_initrd_flash.sh jetson-agx-orin-devkit nvme0n1p1

initrd_log_flash.txt (50.5 KB)

This command will not work.
You should do

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 \
  -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/generic/cfg/flash_t234_qspi.xml" \
  --showlogs --network usb0 jetson-agx-orin-devkit internal

@DaveYYY While running minicom, I get this below Part error

But if I go to boot manager and change the boot from emmc, it works.

There seems to be a bug that NVMe and USB cannot be mounted after building with RT patches enabled; we are investigating it.

@DaveYYY Its not even that, flashing this causes a lot more. Please see my other post attached,

Is there a timeline for this or rolling back to JP5 would be good idea ?

36.2 is only a developer preview version, so it’s expected that there will still be many bugs.
If 35.4.1 works fine for you, then please keep it.

1 Like

@DaveYYY This got be a joke as jetpack 5 with RT patch does not mount either on NVME and display stays black as mentioned in Note but at least its mentioned, my sincere request to Nvidia if something is not confirmed don’t publish in manual, just because you are late and have to publish to server its not worth to even have JP 6 out even after delay because its not usable for some developers and same for JP5 with RT patch. I read lot of posts where
people have posted the same issue without success and some of the moderators have mentioned that its low priority.
JOKE OF 2023 this is, for developer on other side building top level architecture.

In my case jp6 baseline is very good. But trying to upgrade kernel I see that the out-tree drivers are outdated…

@DaveYYY I have fixed most of the kernel bugs on RT patch JP6 side, is it possible for you to realistically check on flashing on NVME side of things in a timely manner ?
OR
Can you also please check on JP5 RT patch flash on NVME, which ever route might be easier thinking JP5 ?
Note if you wanna forward to your internal team:

With RT patch, gnome-control-center crashes, reason being brightness control module (there is a simple fix to initialize brightness to 0 when the jetson boots up) for 5.15 kernel which also leads to gsd-power to also crash which ultimately does not let “Settings” tab to open unless re-installed and reboot the system but then too again when you press on “Power” tile in settings the system crashes again. This is known issue in 5.15 kernel.

YES. We have identified the root cause of this issue.

The issue is that for JetPack 5, NVMe and USB drivers are integrated into NVIDIA’s downstream kernel, and built directly into the kernel image; for JetPack 6, in order to be compatible with upstream kernel, a lot of drivers are pulled out of kernel source, and built as kernel modules.

Therefore, initrd must contain these drivers for NVMe/USB to be mounted as rootfs during booting, but initrd does not get updated during kernel building/flashing, and the kernel suffixes (5.15.122-tegra/5.15.122-rt-tegra) do not match, so drivers cannot be loaded.

A temporary solution would be to copy and unpack the initrd image to the host PC used for building kernel, change the folder name, replace those .ko files with the correct one, pack the initrd image again.

I don’t see any issue with RT patch + JetPack 5.

This is not related to flashing, so please file another topic.

2 Likes

@DaveYYY Rookie question, when you say image, do I get from below ?
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash

OR is it the kernel image ?

OR if you can please direct me towards the path on how to accomplish this, I will appreciate you.

I mean Linux_for_Tegra/rootfs/boot/initrd.

@DaveYYY Thank you, I am trying this process right now.

@DaveYYY looks like there are quite a few .ko files, but just curious is it ok just copy and paste the entire 5.15.122-rt-tegra folder because as you said and made sense that it did not get build and there are driver folders which do not exist so what would you suggest, copy the folder or just make dir as needed and copy the .ko’s in there ? (Image to right is built modules after kernel building and left is unpacked initrd folder)

I think copying the entire /lib/modules/$(uname -r)/ folder should also do it.
(If you think manual copy is too troublesome.)
They don’t make much difference in terms of booting time.
However, it’d be better to strip these modules so the file size gets shrunk largely.

find Linux_for_Tegra/rootfs/usr/lib/modules/5.15.122-rt-tegra/ -iname *.ko | xargs <tool_chain_path>/aarch64-linux-gnu-strip --strip-unneeded

1 Like

I am failing on this step, unless this folder is not supposed to be there ?

Refer to Modifying Jetson RAM Disk.
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/SD/FlashingSupport.html
I believe it’s because you are packing it the wrong way.
I don’t know why you have initrd.cpio.gz there.

@DaveYYY I dont know what is wrong here but my steps are below,

Extracted initrd image and copied the entire folder from usr/lib/modules/5.15.122-rt-tegra to unpacked folder of initrd/usr/lib/modules and then packed with tar -czvf directory_to_tar_in name_of_directory

Then ran the below command to flash,
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1
-c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/generic/cfg/flash_t234_qspi.xml”
–showlogs --network usb0 jetson-agx-orin-devkit internal

and got this error,

Tried couple times, started over with fresh flash to emmc first and then to build kernels with RT patch and try again above steps but same result. You are right, its something to do with packing it as I tried original file to extract and then compress but failed. Is my command wrong above ?

I have no idea who told you to do this.
It’s a cpio archive compressed with gzip, not just a gzip archive. Did you ever read the guide I posted earlier?

find . | cpio -H newc -o | gzip -9 -n > …/initrd

@DaveYYY I am doing a fresh kernel build again now, and sorry no one told me on that process, I was trying to follow the steps above to best of my knowledge which was wrong. So thank you for correcting it, I will let you know shortly here.