Enforce mount, partitioning and other options before flashing the rootFS

[bsp_dev]

Hi,
I wander if there is a way for configuring inner sub partitions, mount filesystems and set mount options on my RootFS before flashing it to the NX.

The goal is to mount filesystems with different mount options, some of them on seperate partitions and doing all this before flashing the nx instead of doing this only after Ubuntu setup.

Same question regarding tools and daemons configurations such as USB guard, appArmor etc. Can I install and configure them in my rootfs before flashing the NX so after flashing the target they will be ready and configured (saves time to configure them again on another target)

Thanks

[linuxdev]

Most of what is in “Linux_for_tegra/rootfs/” is an exact copy of what gets flashed. Depending on your flash target some of “rootfs/boot/” gets changed, e.g., kernel and device tree and extlinux.conf. Everything else is more or less an exact copy.

For example, if you create a mount directive in “rootfs/etc/fstab”, then flashing will result in that working on first boot (be careful to use a correct mount option). I add home directories and ssh keys and some network setup before I flash.

Incidentally, if you set up a Jetson the way you like, including things like package updates, you can clone this and use the clone either directly or as a replacement for “rootfs/”.

[bsp_dev]

Thank you for your reply.

What about partitioning? Can I do it from the host before flashing the target? (partitioning the internal emmc or an internal nvme in my custom hardware). For example I wish to mount /home on a separate partition

[linuxdev]

TLDR; You can edit content in “rootfs/” directly most of the time, and only some “rootfs/boot/” content is at risk of modification during a flash. Examine “tools/l4t_create_default_user.sh” to see how QEMU can run Linux commands in that rootfs which will enter the final partition. Do not treat an SD card model with any of this because it isn’t the same (creating SD card content is far simpler/easier, just ask).

---

You can’t directly partition from the host, at least not in the usual sense (a recovery mode Jetson is not a block device like a USB thumb drive). Are we talking about eMMC models? If so, then an .xml file can change sizes…but the only one you’d change size of is the rootfs; other partitions (which are only in QSPI if this is an SD card model, which you can’t get to) would only be customized if you’ve changed size requirements of boot content, e.g., you’ve custom built some boot binary and it is now too large for the original partition.

Before I mention any of that, you might find it useful to understand putting that partition onto eMMC (we’re ignoring dev kits with SD card boot because creating an SD card is different from creating boot and rootfs onto eMMC). If this is an SD card model, then you’re greatly complicating your life when something much simpler works.

Any time you flash, a file is generated (on the host PC) at “Linux_for_Tegra/bootloader/”. The file generated is “Linux_for_Tegra/bootloader/system.img.raw” (slight inaccuracy for simplification: It is actually system.img, then this gets moved to system.img.raw; finally there is a step to create a system.img from system.img.raw). This is a 100% exact bit for bit blank file (at least at the start) the size of the rootfs partition. If the rootfs is 28*1024*1024*1024 (28 GiB) bytes, then that is the size the partition must be specified as during flash.

The system.img.raw is then covered by loopback so commands operating on partitions can treat the file as a partition. That loopback device is formatted as ext4.

Then, depending on target requirements (such as an Orin kernel being different than an Xavier kernel, or a custom boot sequence differing fromIf one were to flash on command line, and if there is a default one), some boot related content is copied into “Linux_for_Tegra/rootfs/boot/”. Following this, the entire content of rootfs/ is copied into system.img.raw. This is the “raw” image and can be used to flash.

A “sparse” image is created from “bootloader/system.img.raw”, which creates “bootloader/system.img”. This file, rather than being the size of the partition, might be considered “compressed” (it isn’t really), whereby instead of actually having the blank space it simply marks blank space and doesn’t fill it in. The system.img.raw and system.img contain the same information and files, but the sparse image is much smaller (the exception being that as the filesystem fills up, the sparse image grows and approaches the size of the raw image). The only real purpose of this is to save space, although not on the host PC; during flash and creation of rootfs it simply does not require nearly as much time to flash a sparse image compared to a raw image. However, sparse images can’t just be loopback mounted and manipulated, they can more or less only be used for flash. Raw images are much more useful if you don’t care about time to flash. The host PC does not remove the raw image after sparse is created.

So long as there is a system.img, then that is what gets flashed. If that is the raw or sparse version it won’t matter, the result, other than time to flash, is the same. If we are manually flashing though, and we have a custom partition size which is not the default, then we must name the partition size in the options (or for automated flash, in the correct .xml file). The .xml file itself is usually generated from other files depending on the arguments to flash, but examine “bootloader/flash.xml” to see. Manually flashing is simpler. An example, if our rootfs partition is customized to 16 GiB (16*1024*1024*1024 bytes), and if the target is deIf one were to flash on command line, and if there is fined by “Linux_for_Tegra/jetson-xavier-nx-devkit-emmc.conf” (target “jetson-xavier-nx-devkit-emmc”):

# Do something to generate "bootloader/system.img".
# Notice this is the .conf name without .conf suffix:
sudo ./flash.sh -r -S 16GiB jetson-xavier-nx-devkit-emmc mmcblk0p1

Note that mmcblk0p1 is an eMMC target. We can’t equate SD card models with eMMC models so far as rootfs goes. The “-r” says to "skip building system.img and use what is there. Even if you didn’t customize, if you flashed before, then this is useful to save time not rebuilding the same image. The “magic” here is the “-S 16GiB”; this specifies how much space is to be reserved in the partition layout for the system.img.

There are many ways to build a system.img. They are like algebra, and have interchangeable quantities. Command line flash will install exactly the right thing if partition sizes are specified correctly. Beware that saving and moving around those images is quite slow since the smallest is probably 2.5GB for a sparse file, and much much larger and slower with a raw file.

You can create a raw and sparse via clone of an eMMC model. You could then flash that over and over. You could loopback mount and examine or modify the raw clone at any time. You can loopback mount and examine the raw generated file as well. By flashing on command line and saying to reuse that file you can be guaranteed your rootfs is an exact copy of that down to the last bit.

Since covering a file with loopback can make it appear to be a block device, you could use an arm64 emulator (e.g., QEMU is already present if you’ve already flashed once) and run commands on it, e.g., a command to add a user name, or update packages. An example to emulate if you want to do this already exists as “Linux_for_Tegra/tools/l4t_create_default_user.sh”.

What “l4t_create_default_user.sh” does is to use QEMU and to run commands to create your default user and complete first login directly in “rootfs/” (which implies any system.img it creates is also completed). One could use that as an example and complete other commands on the rootfs (though I don’t know about adding networking to the QEMU; I’m unsure if that is present or not, but some commands, e.g., “apt-get”, might need extra environment setup…you could always work on an example, and if it fails, copy it to the forum and ask about it).

If the content to be edited or changed does not need an arm64 executable, and is just a file copy, then you directly do that. For example, my home directory is already set up, including ssh, which I did by simple file copies (or rsync) into the rootfs/. Various network settings and host alieas are there the instant the flash completes.

Side mention: If you edit “/etc/skel”, then when you create a new user with defaults, this is where the defaults come from; if you create a lot of users consider poking around there to see how default user home directories are set up.

There are some “gotchas”, whereby if you change the size of one partition it might mean there isn’t room for the other and it’ll happily do the wrong thing. Replacing some content also requires doing so with a signature, but you don’t need to worry about that with a rootfs. Before you start, if you already have a “bootloader/system.img.raw”, do yourself a favor and record the exact byte size of that before you start; this is what was used last and is likely the default.

Specific use-case examples of what you want to do, combined with the exact model you are working with (third party carrier boards matter greatly, and so does eMMC versus SD) could narrow down an answer to something more specific.

[bsp_dev]

First, thank you very much for your detailed answer.
I admit I didn’t understand some of the things you wrote here so , by your permission, I will be more specific with my question while repeating parts of your answer to make sure I understood correctly (maybe it will serve others)

0. Setup:
a. Xavier NX emmc model
b. Additional Xavier NX emmc model connected to an internal NVME on a custom board (not related to sec. a) The rootfs needs to be on the internal NVME.
c. Currently JP 4.4 and JP 5.02 in the near future

1. General goals:
a. Build a minimal, custom rootfs (not the sample one) , probably using buildroot.
b. The rootfs will include as much as possible desired packages and settings before flashing the actual target. for example:

• Users, passwords, permissions
• Services and their configurations such as USBGuard , AppArmor , PAM configuration , Auditd, rsyslog and more…
• Network settings such as firewall, ip address, etc…

c. Create separate partitions for user files in order to enforce mount options and limitations, plus for services such as /var/log for auditd
d. encrypt the rootfs
e. create a second minimal rootfs in case of any failure for recovery purposes

2. My conclusions so far + questions:
a. minimal custom rootfs:

• Its possible to use a buildroot rootfs without any problem.
• question: I need to understand how to add needed drivers and libraries in order for the system to function properly.

b. The rootfs will include as much as possible services and settings prior to flashing:

• I can easily apply various settings by copy the relevant file into the rootfs directory.
• question: I understand from your answer that I can use QEMU to install services (and configure them) as well. Can you please elaborate (briefly) how it works? does it creates a VM which uses the rootfs directory as its own, install to it whatever I want and then leaves the rootfs ready for flashing the target including all the installed services? if so, I can basically put there what ever I want?
• The reason I prefer not to clone an entire rootfs image is because I wish to stay versatile with my rootfs and not be bounded into specific hardware (you said that during flashing process some of the /etc/boot files are modified)

c. Create separate partitions prior to flashing

• I understand from your answer that it cannot be done. I have to flash the target once, manually create the partitions that I want and only then mount and copy files to each partition after each flashing.

d + e. I don’t know how to do that but I didn’t study it yet since I’m busy with earlier steps.

[linuxdev]

Some notes, not in a particular order, so I won’t forget while I read:

• You have an eMMC model. This implies most boot content must remain on the eMMC, and various setup/install approaches can boot to alternate media. It is, however, simple to boot to eMMC, optionally mounting extra partitions of other content on various non-root mount points. Not really what you are asking about, but useful to know when working on the filesystem.
• If this is a third party carrier board (which seems likely since it is an eMMC model of an NX module), the implication is that you most likely must use flash software from the third party carrier board vendor. This is not always the case, but if it is not, then the manufacturer would specifically state this.
• The one unit you know is a custom board, but I suspect the other one is also custom (any carrier board not built by NVIDIA could be termed either “custom” or “third party”; electrical layout determines whether or not software has to change to accompany this, and most often it does have to change).
• Two approaches to customizing any rootfs (I’m not suggesting you switch methods, this is to illustrate flexibility):
  • Install normally, customize, remove packages, and then clone. Use that for rootfs.
  • Build your content as you are doing, and put it in “Linux_for_Tegra/rootfs/”, but make sure “sudo ./apply_binaries.sh” still works (this is the mandatory content; it’d already be there from a clone). Optionally, beware that this might include first boot setup, but if you’ve added an account already, then this might conflict in part; thus would suggest not adding an account yourself and instead using “sudo ./tools/l4t_create_default_user.sh”; then installing the home directory of that user.
• I typically use “l4t_create_default_user.sh”, and then add my own home directory, along with overwriting several known “rootfs/etc/” files set up for my taste, adding ssh keys, so on. When I boot the system “just works”.
• I suspect that if the “l4t_create_default_user.sh” works on your custom rootfs, then chances of success go up dramatically.
• Beware that during flash, unless you use a clone, then some of the “/boot” content will be overwritten based on flash parameters, so you need to know that most, but not all of your rootfs will be exactly what you created.
• I can’t tell you all of what you will need for the actual rootfs content if you don’t start with a full install and cut content followed by clone, although some people here can answer such questions. You’d probably need to start a thread on specific issues as they occur.
• Part of booting is always on eMMC of those models, so even if you customize rootfs correctly, then you must still flash such that eMMC is set up for your custom content.

Part 2, Qs:

• Adding drivers and libraries is kind of a quite big (but not too complicated) question. Drivers are normally from the kernel configuration, although there are on occasion some user space drivers (such as the NVIDIA GPU driver having a user space component which plugs into the X server; indirectly, much CUDA software can depend on this).
• Specifically regarding the NVIDIA GPU driver, you need to preserve the X server ABI (look for ABI in the “/var/log/Xorg.0.log” to find which ABI currently runs). Without that GPU is more or less useless. You can still use framebuffer video, but no acceleration. So this X server release must be locked in whatever package system you use. Note that the ABI is the “Application Binary Interface”, which is like a specific API version you might think about when linking libraries, but it is for binary object loading.
• For drivers in general though, on a working system, examine the output of “zcat /proc/config.gz”. That’s the exact configuration of the running kernel as reported by the kernel itself. This is essentially the “.config” which was used during the build of the kernel, minus the “CONFIG_LOCALVERSION”. You must still manually “do whatever the right thing is” with “CONFIG_LOCALVERSION”, which in turn determines where modules (which are loadable drivers) are searched for (get that wrong and only the integrated features will work; all other drivers would be missing).
• Libraries depend on whether you use a package system. Are you using a package system? Can you just create packages and add them to your “rootfs/” content via QEMU? Note that the script “l4t_create_default_user.sh” is a very good scripted example of how to use QEMU to make changes to the “rootfs/”. You can edit this if you need a package system.
• The “l4t_create_default_user.sh” does more than just add an account, it also completes the first boot setup content, but this is already all you need for adding users. Simply copy that script to some new name; I’d suggest two versions if you need both admin and non-admin users, e.g., “l4t_create_custom_admin_user.sh” and “l4t_create_custom_user.sh” (these would be edits of the original script).
• Just to overemphasize: Check the “l4t_create_default_user.sh” script. It is human readable bash script. If you’ve flashed normally, then the QEMU part is already set up and all you have to do is edit that script (copy to a test version and see if you can edit to do something simple and minimal, e.g., create an empty file with “touch”; then do something more interesting, see if you can edit to read “dpkg -l” to read the package database, or run “ldconfig -p” to see the linker path using the arm64 tool).
• If libraries are loose files, just put them in the correct place and make sure the linker path looks there. If it is a standard location, then it is quite simple. You can see an existing linker path with “ldconfig -p”.
• Any service or configuration anywhere except “rootfs/boot/” is just an edit (some of boot content is safe from overwrite, but you’d have to know by watching a logged flash to see). Copy the file there, and that’s what you get upon flash.
• Creating partitions on eMMC is the realm of the XML files in the flash software. I can’t help much with that, I have not customized partitions. However, there are many people you could ask if you have a specific change, plus there are several posts on the forum about partitioning.
• Partitioning on external media is probably done manually directly to that media, but I can’t say for certain (I’ve not worked with external media).
• I can’t answer about alternate rootfs; there is the A/B redundancy you can set up (I don’t know enough to say how), but that’s different than an active second partition.
• Some of the major files of “/boot” (which on the host PC are at “Linux_for_Tegra/rootfs/boot/”) are modified during flash, including the kernel and device tree and extlinux.conf. Nothing in “/etc” (at “rootfs/etc/”) is altered and anything you edit there will remain edited if you change it. It is quite valuable to flash once on command line and log the flash just to see what files get copied where…that’s your bible of what changes upon flash. Save that log and don’t lose it even if the flash was not something you’re actually going to use; the flashed device does not even need to boot, the point is to see what files get copied to where during flash.
• Note that when using QEMU it will essentially do a chroot (the filesystem will change root to the “rootfs/” point, and “/” will then refer to “rootfs/” until the environment is existed; tools used while in QEMU will be arm64 tools, either directly from the “rootfs/”, or from a QEMU cross tool. Try the l4t_create_default_user.sh just as an exercise.
• I don’ t know that QEMU is “exactly” a VM, but more or less it is. It runs in an amd64 environment, pretends that the root of the filesystem is some new chroot location (“Linux_for_Tegra/rootfs/” if you use l4t_create_default_user.sh), and can call and use that new rootfs location as if it is a running computer. For example, a useradd program would run based on the one in the rootfs, and if that program is missing, then it would fail; a full VM I think adds its own environment, and you could in theory add the rootfs in the path, but the default search path might not be just a simple chroot to a new location. The package manager tools within “rootfs/” would be the ones used if you used an apt or dpkg command. The linker from rootfs/ would be used if you ran an ldconfig command. All of those commands would think that “rootfs/” is “/”. rootfs/ is indeed its “root of all things”. It is similar to if you were able to log in directly to that location and forget the host PC exists (but limited in scope).
• Remember that you will always be limited to the boot content from NVIDIA (although it is customizable), and you must use the Xorg server of the correct ABI if you wish to use the GPU.
• Incidentally, I personally modify ssh server and networking, and sometimes services, via directly editing content in “rootfs/” prior to flashing. This works great. The time when QEMU matters is when you need an arm64 version of a program to run, e.g., the package manager (adding packages involves updating a database and is not a simple file copy).
• Just to emphasize, partitioning is a topic of the XML configuration files and not part of the rootfs. I don’t know enough to advise on that. Those changes would be for eMMC content, and you must have some eMMC content even if rootfs is not on eMMC. There might be some use in the XML files reflecting external storage device layout, but I don’t know.
• All partitions except rootfs preexist as binary content, which is then signed before flashing (without signature boot will fail, although basically the default is a NULL signature; even a NULL signature is a signature). rootfs is not signed, and is the only one which is generated prior to flash. Those signed partitions always go on eMMC, you cannot put them on external media. Creating the space for a partition is done in XML files, while flashing the signed binary content creates the actual partition.

I strongly suggest keeping threads separate between partitioning and the rest of your project, e.g., creating partitions is very different than questions on creating a custom rootfs. Different people would read those threads. I do think that external media partitions will likely be manual.