Flashing /root to eMMC and /home to SSD

EDIT: please see my first comment for a better question formulation

Hello,
I purchased a 3rd party board for Jetson Nano A206 Carrier Board for Jetson Nano/Xavier NX/TX2 NX with compact function design and same size of NVIDIA® Jetson Xavier™ NX carrier board - Seeed Studio
I managed to flash the system.img with NVIDIAs Sample Root Filesystem to eMMC following this guide: A20X Carrier Boards Serials Getting Started - Seeed Wiki I used the command for creating the image:
sudo ./flash.sh --no-flash --sign jetson-nano-devkit-emmc mmcblk0p1
and command for flashing it:
sudo ./flash.sh -r jetson-nano-devkit-emmc mmcblk0p1

However, in order to save eMMC storage, I would like to flash the rootfs onto 128GB NVMe M.2 PCle Gen3x4 2280 Internal SSD 128GB NVMe M.2 PCle Gen3x4 2280 Internal SSD - Seeed Studio, and flash the Tigra_for_Linux OS onto the eMMC. How should I go about it?

I couldn’t find a topic discussing flashing Linux and rootfs to different storage units.
I would appreciate any help, since I am quite a beginner :)

What is the definition of your “OS”? Is it just the bootloader? or the bootloader +Linux file system?

The linux file system includes the rootfs. So it may not be separate.

I just had a conversation with a more experienced colleague, which helps me to reformulate my question better: I would like to have the root / on the eMMC, and the /home in the M.2 SSD, since it has a higher capacity. I already know how to mount the SSD automatically during 1st boot (I have to edit Linux_for_Tegra/rootfs/etc/fstab). Now I would like to know how to move /home to the M.2 SSD and populate /home with a custom file system, all during the flashing process. That way our robot production can just flash the Jetson Nano and mount it straight to the robot without having to connect the monitor and manually install stuff.

This is quite easy to do, but is not done during flash. Any Linux doc will describe this and it isn’t any different for a Jetson. This is how I manage my host PC. I’ll describe below, starting with an example of certain behavior.

For the sake of argument let’s say “/” is “/dev/mmcblk0p1”, and that you also have an empty partition “/dev/nvme0n1p1” (but formatted for ext4). Also, for the sake of argument, let’s pretend your user name is “ubuntu”.

Before mounting anything special, consider that you have content in “/home/ubuntu”. If you were to manually mount over this, then you would:

sudo mount /dev/nvme0n1p1 /home
cd /home
# If you run "`ls`" or "`du -h -s .`" it would show no content.

Now if you leave that location and umount:

cd /
sudo umount /home
# Now cd back:
cd /home
ls
du -h -s .

All content would show up again. Whenever you mount a partition over an existing location, that content becomes inaccessible on the original partition, but that content is still there unharmed. You could “move” that content onto the NVMe, or you could simply “copy without changing the original data”, and once mounted to “/home”, the result would look the same.

For the case of not removing the old data, this could serve as a backup, or as an emergency content should the NVMe be removed. Leaving the old data in place though would mean the eMMC has used space which cannot be accessed (at least not while the NVMe is mounted). If you don’t need that extra space, then I’d advise leaving it there for emergencies or utility.

There is nothing you need to do which is “special” for first boot, although you would still want to customize “/etc/fstab”. However, there is an important point about the different mount options: If your fstab mount specification does not make this “optional”, then it means boot will fail by locking up at the point in boot where that partition mounts, and it will wait forever. You’d end up flashing again to correct this, or else adding a partition matching the mount specification. If the mount is optional, then boot will continue without the mount if mount is not possible; even so, the mount will do as you wish if it is possible to mount. Mount options matter.

Let’s say you have blank partition, and you want to truly mv this rather than cp. You’d mount on an alternate mount point, and literally use the “mv” command. Example:

# Temporary mount:
sudo mount /dev/nvme0n1p1 /mnt
cd /home
# Copy to temporary mount:
sudo mv * /mnt
# Be certain there are no "`.name`" "dot" files still present...the "`.`" and "`..`" of course
# don't count. Being "`/home`" there won't be any, but similar copy from some other
# location might have such an issue.
cd /
sudo umount /mnt
sudo mount /dev/nvme0n1p1 /home
ls /home/

On the other hand, if you want to copy and not truly move, then use a tool designed for this, e.g., rsync. Example (note that there are many ways to use rsync, and I am throwing in the proverbial “kitchen sync” with progress logs, so on):

# Same temporary mount:
sudo mount /dev/nvme0n1p1 /mnt
cd /home
# Now it differs:
sudo rsync -avcrltxAP --info=progress2,stats2 --numeric-ids --exclude '.gfvs' --exclude 'lost+found' /home/* /mnt
# Same warning about "`.name`" "dot" files...the "`.`" and "`..`" don't count. "/home" won't have any
# dot files, but other locations might.
cd /
sudo umount /mnt
sudo mount /dev/nvme0n1p1 /home
ls /home/

rsync” can do the right thing with symbolic links, device special files, named pipes, so on, in ways which “cp” might do incorrectly. “mv” is not generally an issue because it relocates files rather than copying contents.

About permanent mount and “/etc/fstab”: You can specify what to mount either via a device path, e.g., “/dev/nvme0n1p1”, or via a UUID. I prefer UUID because it can be used on any device, but there is a learning curve involved. If you want to know about UUID just ask, but for the moment I’ll go with the simpler device path. Also, if you mark the device as “ok to fail”, then boot would continue even if the device is missing or fails. Should you have created a copy instead of a move, then the original content will be there to provide an environment for your user during the missing/failed mount. Summary:

  • Optional mount means it can fail and you won’t need to flash since boot won’t lock.
  • Copy instead of move means that if mount fails you have a backup environment.

Specifically, a line in fstab will look something like this for the most basic case (where failure will break boot):

# field1         field2  field3 field4    field5 field6
/dev/nvme0n1p1   /home   ext4   defaults  0      2

Btw, you can put as many spaces as you want between fields in fstab for clarity.

Note that “defaults” is equivalent to a comma-delimited set of these options:
rw,suid,dev,exec,auto,nouser,async

You can tell it to allow errors, and thus continue to boot if either the partition is missing or somehow failed. That is the “nofail” option. An example with all else being default:
rw,suid,dev,exec,auto,nouser,async,nofail
(notices “,nofail” on the end…no spaces between any option)

For debug purposes, while experimenting with partition mounts, it is sometimes useful to know about the “noauto” option. Normally you could not just “mount /home” as it is missing options like which drive is there. Should you have this entry, then it would not auto mount, but it would do exactly the right thing if you “mount” or “unmount” either “/home” or “/dev/nvme0n1p1”:
rw,suid,dev,exec,auto,nouser,async,nofail,noauto

During flash, if using defaults, the content of “Linux_for_Tegra/rootfs/” is used almost verbatim to create the image to flash. “Almost” because some of the “/boot” content will be altered depending on boot options, e.g., the Image and extlinux.conf can be altered, but the rest is left alone. So if you were to add this line for testing, then after boot “/home” would “just do the right thing” if available:

/dev/nvme0n1p1   /home   ext4   rw,suid,dev,exec,auto,nouser,async,nofail  0  2

You can read about fields 5 and 6 in “man fstab”. These are related to what backup software will do, and the order in which done. One would for example not want an fsck.ext4 on both “/” and “/home” at the same time should something go wrong…you’d want this to be sequential. If “/” is order “1”, and “/home” is order “2”, then it’ll first fsck/” if needed, and then fsck/home” if needed. Thus the “1” or “2” in the last field.

1 Like

I played more with UUID, and if I understand it correctly, in production I will not be able to see the UUID before flashing, booting, and running sudo blkid. And thus I would not be able to add the correct UUID to the /etc/fstab in the rootfs used for flashing. On the other hand, putting /dev/nvme0n1p1 in /etc/fstab before flashing should be possible. Is my hypothesis correct?

Mostly this is about eMMC flash, but It is easiest to use “/dev/somethingWithAName” during flash (versus UUID), but it is not correct that you won’t know the UUID. It is just a pain to get it relatively speaking. If you cover the raw version of the flash filesystem image (“Linux_for_Tegra/bootloader/system.img.raw” during normal flash, or the raw version of whatever you flash when “reusing” an image), then you can get the UUID from that. As long as you reuse that image the UUID shown from the blkid command of the loopback device will also exist on the system after flash.

On the other hand, people don’t necessarily flash the NVMe when flashing the Jetson, and probably add that content outside of flash (the flash would then change what the Jetson searches for first). When and how do you actually add content to the NVMe? That is the key to answering whether UUID is easy to use or not (but it still is more likely that it is “easy” to just use “/dev/nvme0n1p1” instead of UUID).

That’s another question I am trying to figure out answer to. Our software team has a codebase that includes docker and runs on Jetson Nano that controls the robot.

Ideally, I would like to add the content (latest stable version of our codebase) in the same host script that calls also sudo ./flash.sh -r jetson-nano-devkit-emmc mmcblk0p1, so that the robot production team doesn’t have to log into the Jetson Nano after flashing it, but can simply mount in on the robot and test it.

I managed to figure out how to modify /etc/fstab to auto-mount the NVMe on every boot, but I don’t know how to automate adding content to it.

Whatever is on “bootloader/system.img” during flash ends up on the final flashed system. If that image is from a clone which had updated packages, extra software, and a set of login credentials, then this is what the Jetson will have without adding anything else.

system.img” can be raw or sparse format. Sparse cannot be loopback mounted, but flashes faster. Raw can be loopback mounted, modified, so on. Raw can be used to create a sparse using mksparse with a 0x0 fill byte.

This of course matters more for eMMC flash. You can use simple dd tools to copy an NVMe or SSD partition rather than cloning.

How about moving /home from the eMMC to the NVMe, as described here? Move Home Directory to another dedicated Drive/Partition in Linux - .matrixpost.net

You don’t need parted to create a partition on an empty NVMe, but it could work even if starting empty (it’s just more work, although if you don’t know the base commands, then it might seem like less work). Do be sure to use UEFI partitioning and not old style BIOS partitions (e.g., “fdisk” creates old style partitions, and “gdisk” creates newer UEFI partitions and is preferred…something like parted…or better yet, gparted…would usually preserve whatever partitioning scheme is already there even if it is the less desirable BIOS style unless you tell it otherwise). Favor gdisk over fdisk, and gparted over parted.

The filesystem type used on “/home” should be ext4. There are cases where an SD card might boot with type VFAT for various reasons, but normally ext4 should be used. The tool “mkfs.ext4” is just “mkfs” with ext4 options, so it is a nice shortcut when making the filesystem on the partition you will use for home.

As for mount point you want any automatic mount of the new media to avoid automount in the “/media” location. The rules to add to “/etc/fstab” should normally take care of that.

The use mentioned of rsync to perform the copy is good advice. You can move the old home to a backup location, or just leave it there since its presence will be hidden while the NVMe is mounted over it, and would be a useful backup for cases when the NVMe is missing.

I did not actually go through that post closely enough to try it out, but basically the above notes would apply and otherwise that URL seems ok.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.