Is that 128 GB SD card USB? Or is it part of the SD card which plugs in to the module itself (not the carrier board)? There are documents on how to use the full size of the SD card if it plugs into the module itself, and you could avoid what follows and simply make the whole thing available. Someone else can explain this, but the documents for your specific L4T/JetPack release tells you how to make the entire SD card available. What follows is if your SD card is separate from the one providing the rootfs, e.g., a USB external SD card.
Understand that requirements for success of something not used during boot will differ from requirements of something that is used during boot. “/var” must be present on the rootfs during boot. An example of two places which tend to consume a lot of space, and which could be missing during boot and not cause boot failure, are “/home” and “/usr/local”. “/var” must never be missing during boot.
Normally one would mount a partition at the correct location instead of using a symbolic link. They can be equivalent, but you can’t control order of access of a symbolic link during boot the way a partition mount can be.
In your case the rootfs is no longer valid and needs to be repaired or recreated. The QSPI won’t care about “/var” missing since “/var” is not a boot stage software. If you still have the “/var” content on something else whereby you can mount both at the same time on a host PC (e.g., two SD card readers or two separate partitions of a single SD card), then you might be able to just fix it (but you’d have to be careful to preserve permissions and any device special file or named pipe).
What I’ll describe is how people would normally offload just “/usr/local”. The same thing applies to “/home” (I do that on my host PC; when I upgrade my Ubuntu release I just remount my home content; or if I multiboot between Ubuntu 18.04 and 20.04, it just remounts “/home” on the correct rootfs and I get the same home content regardless of which I boot to).
When you first boot the Jetson will mount “/”, which is the root of the filesystem (thus the rootfs). Assume you have content in “/usr/local” (see “sudo du -h -s /usr/local” to find out how much is stored there). I’m going to assume your SD card partition is “/dev/sda2” (it might be quite different, adjust for whatever partition the extra content is on).
If you do this:
sudo mount /dev/sdb2 /usr/local
…then all previous content in “/usr/local” no long shows up (it’s still there, but you can’t access it).
You could then:
- Either
sudo umount /usr/local
- Or
sudo umount /dev/sdb2
…in which case the original “/usr/local” content would show up again.
If you were to have a duplicate of the original “/usr/local/*” content on “/dev/sdb2”, and then you mount “/dev/sdb2” on “/usr/local”, you wouldn’t be able to tell the difference. However, any changes made would go to the SD card, and upon umount, the content would revert back to the original “/usr/local”.
Mounting a partition at a location hides original content and makes the mounted partition become the memory storing it. You could (if done properly, e.g., via rsync) copy the original “/usr/local” to a temporarily mounted “/dev/sdb2” (you could mount it in “/media” or some temporary location), you could then delete the original “/usr/local”, saving space, and then mount the sdb2 partition on “/usr/local”, and you’d have all of your original content available. The space would be on that new partition.
Mount order matters. You have to mount “/” first. This is the rootfs, and you don’t want to mess with that. If you then and only then mount “/usr/local”, it would do what you want. As another example, suppose you have a copy of your original “/home” on the SD card. The system boots, and then you mount “sdb2” on “/home”; if you had properly synced the old data onto sdb2, and then deleted the original “/home” content, things would work exactly as you expect. Now let’s say you also want yet another partition to save space on “~/nvidia” (I do this). I’ll pretend the device partition being mounted is “/dev/sdc1” (I’m making that up as an entirely new drive’s first partition formatted to ext4), and that my user login name is “ubuntu” (which means “~” is the same as “/home/ubuntu”). This would work after boot:
sudo mount /dev/sdb2 /home
sudo mount /dev/sdc1 /home/ubuntu/nvidia
sudo umount /home/ubunt/nvidia
sudo umount /home
(note that if you “cd” to a directory on one of those mounts, then you cannot umount until you exit that location…attempting to do so would reply with the error that the drive is busy)
It is important to know that you can automate mount, but if you do so in such a way that mount is mandatory, then not having that drive would cause boot to halt and wait forever. Since you have an SD card partition, you might be tempted to remove the SD card, but done incorrectly, this would make boot fail until the SD card is back in. The place to automate mount is the file “/etc/fstab”. This is also where you must be careful, or else boot will fail.
I assume any partition listed is formatted as ext4, not VFAT or something else. Assuming your mount point is “/usr/local”, and assuming your partition is from “/dev/sdb2”, then this would automatically mount the partition and not fail if the SD card is missing (this is a line in “/etc/fstab”):
/dev/sdb2 /usr/local ext4 defaults,nofail 1 2
The “1” just says this is something that would be “dumped” during a backup (it would be backed up), and the “2” is the order of checking filesystems. The rootfs is “/”, and this is the “1” even if you don’t see it, and since the SD card is mounted after rootfs, it is “2” instead of “1”.
If the partition were mounted on “/home”, and then another is mounted on “/home/ubuntu/nvidia”, you’d have to increment the “2” of "/home/ubuntu/nvidia" to a "3" (rootfs is 1, home is mounted on that as the second mount, so it becomes 2, and /home/ubuntu/nvidiais third since it mounts on/home`…they have to be checked in order so the most root of the tree is checked first).
The “nofail” is what makes it possible for boot to continue even if the extra SD card partition is missing. The “nofail” cannot be used with mandatory partitions, e.g., it can’t be used with “/” or “/var”.
If you wanted to prepare partition “/dev/sdb2” to become a replacement for “/usr/local”, then you might start with sdb2 mounted somewhere temporarily, e.g., maybe it automounted on “/media/1000”. That’s ok as a temporary mount point.
But is it ext4? If you’ve never formatted it, then it is more likely VFAT. You could delete content and reformat it like this (be very careful to use this only on a partition which is not in use; use umount first):
sudo umount /dev/sdb2
sudo mkfs.ext4 /dev/sdb2
# Any temp location works:
sudo mount /dev/sdb2 /media/1000
# This preserves everything 100% correct while copying from "/usr/local" to the SD partition:
rsync -avcrltxAP --info=progress2,stats2 --delete-before --numeric-ids --exclude '.gvfs' --exclude 'lost+found' /usr/local/ /media/1000
sudo umount /dev/sdb2
# Now edit /etc/fstab as mentioned above.
# Now either reboot or simply test it out:
sudo mount /usr/local
If the above works, then the fstab line is correct, and the content in “/usr/local” will look just like it did. The original “/usr/local” content is still there, and if you test for some time and find it acceptable, you could delete the original “/usr/local” content.
I will probably be gone for about a week so I won’t be much help for some time, but if you are just using the original SD card and want to make full use of it, then as mentioned at the start, you don’t need to do any of what I just mentioned and someone else can help expand to use the full SD card. It is useful though to make this method known since other people will run into the situation.
