System RAW image L4T 32.x.y l4t_initrd_flash.sh

Dear nvidia L4T support:

Please explain the path forward to support large NVME SSD drives in the L4T framework.

Background:
The l4t_initrd_flash.sh script creates a RAW system image on the host system equal in size of the total of all partitions specified to the flash_l4t_nvme.xml, less the size of the UDA sector. Put another way, the “-s APPSIZE” passed to l4t_initrd_flash.sh sets a floor for the RAW image size. This is discussed somewhat in this post:

The default “num_sectors” flash_l4t_nvme.xml is 121208832, ~57 GB, meaning a 64 GB NVME flash layout, with an APPSIZE of 44GiB. 64 GB M.2 NVME drives are no longer available at retail.

Question to Nvidia L4T support:
8 TB M.2 NVME SSD drives are now available in the market. 1 TB M.2 NVME SSD drives are now extremely common. The host system has to have sufficient free disk space to build the RAW system image (and perhaps for multiple RAW system images). This is also a deployment/manufacturing problem as the time to flash the target greatly increases as the RAW system image grows ever longer. As NVME SSD drives continue to grow in size, a better target flashing solution needs to provided to nvidia customers. Automatic image resizing on-first boot needs to be supported in the reference rootfs. Or at a minimum a reference guide discussing the issues of / root partition resizing as it fits within the L4T tool framework.

I can’t answer all of this, but it is possible to mount a disk partition on the “Linux_for_Tegra/bootloader/” directory after using rsync (or similar) to populate the new partition. Upon umount of the partition, then the original disk would show up again. Or, better yet, just mount a disk partition for all of “~/nvidia”.

Please manually set the size for now. We will see if any way to improve in future.

@linuxdev. Thanks for responding, Maybe I wasn’t clear in the issues I’d like to see addressed in the L4T tools:
1/ reduce host resources required;
2/ decrease flashing time for production/deployment/re-spins; and,
3/ support / root partition resizing out-of-the-box.

Take a 1 or 2 TB M.2 NVME SDD, bolt it into a Xavier dev target. Build a custom 5.10 kernel and a custom rootfs, using the tools of your choice.Then patch all the issues in Jetpack 5.0 DP to flash the NVME drive using l4t_initrd_flash.sh with an “-s APPSIZE” of 820GiB and correct “num_sectors”. Patching all the Jetpack 5.0 DP issues is left for an exercise for the reader. The system RAW image will be ~840GB. That’s consuming a huge and and unnecessary amount of host disk space as only ~5 GB is really in use. Building a much smaller system RAW image would be a better use of host resources and greatly reduce flashing time – think product deployment and manufacturing, and shortened development re-spins. The / root partition could be resized after flashing to use the balance of the unused NVME disk – the UDA partition. As larger NVME SSD drives are deployed, resizing should be supported out-of-the-box, without changes to to the L4T disk partitioning schema and supporting scripts in the nvidia as-shipped reference rootfs.

Spend half-a-day experimenting with partition resizing on a Xavier NVME drive, then you’ll see what I’m talking about. It can be done quite successfully.

@WayneWWW Thank you.

I’ve thought extensively about your first point. The problem is that a file system has to be created, and this requires a block device. In recovery mode the Jetson is not a mass storage device. Thus, there is no way to perform this operation directly on the Jetson without intermediate memory (the host PC’s disk).

Unfortunately, the recovery mode Jetson is something of a black box, and so unless NVIDIA exposes a recovery mode Jetson as (at least in part) a USB mass storage device (and this might not even be possible), then there is no “simple” solution.

A possible host side workaround would be rather complicated, but possible: Modify the equivalent of loopback program “losetup” (and possibly the kernel) to serialize (a data pipe) losetup where it can take a serial stream (which is what a recovery mode Jetson accepts when writing to it via USB) and apply mkfs.ext4 as if it were a block device, but the stream is used to populate the recovery mode Jetson. This would be a fascinating project (reminding me of the evolution of parallel ATA drives to serial/SATA drives, but for losetup), but not simple, and requiring full access to recovery mode’s internal workings, so it is unlikely “reasonable”.

At least for now adding other storage to the host PC while generating the file system seems to be the only way to go.

For decreasing time, this is why NVIDIA added the ability to flash via sparse file systems instead of the standard raw file systems. Unfortunately one has to first create a raw file system prior to the sparse. The up side is that if you “reuse” the file system, then you never again have to create one. You could in fact have a number of external drives with different file systems and mount the one you want; flash would go fast if you have this pre-created “library” of reusable images. For the “flash.sh” program, this is the “-r” option.

Your last thought on out of the box resizing is something which might be much more possible in the JetPack/SDKM 5 release (or any release using UEFI). Having UEFI actually makes it possible to add intelligent software in the boot stages which would be similar to a bootable install DVD or thumb drive. For this I’d probably need access to the JetPack/SDKM source for it to be useful (there would not be any need to access the proprietary firmware setup occurring in UEFI for setup prior to the standardized parts which are presented in the middle of UEFI boot). UEFI makes a lot of fascinating new possibilities available, but one would need access to the installer software for it to be intuitive.

One variation is based on what you mention about a smaller raw image size: A smaller image could be installed, and then expanded (it would basically be an in-place equivalent to what gparted does, partly in the recovery mode Jetson, and partly in the host side flash software). Big project.

@linuxdev Excellent response. I was able to resize in place using cfdisk with a custom 5.10 kernel and very minimal custom Ubuntu-base focal rootfs (scratch, not from the nvidia reference rootfs). Changes from the default NVME partition layout are required. Resizing should be very scriptable using an init script and sfdisk. And yes, I’ve got an 8 TB platter drive on order for another Linux tower just for this. The entire L4T frame work is a train wreck.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.