Increasing Xavier onboard storage

Hi everyone. During installation of software on my Xavier, OpenCV, Point Cloud Library etc, I am running out of the 32GB onboard storage. I have a 500GB SSD installed as well. Is there any way I can install programs and use them from the SSD rather than the onboard memory.

An example of the easiest way to work with this is to use the SSD for the content on “/usr/local” (which happens to contain CUDA libraries and such, and boot does not depend on that content).

If you have a partition formatted as ext4, then you could copy the existing content from “/usr/local” there, and then mount the SSD on top of this (you’d be free to delete the eMMC content if you are satisfied with the copy).

The same thing can be done with mount of a partition on “/home”, but I tend to leave the original content in place for “/home”…that way if the SSD is not present, then the old “/home” content is still available.

An example might be to use gparted (a graphical tool for partitioning…use “sudo apt-get install gparted” if you don’t have it) to create a partition and format it as ext4. gparted can be used from a different computer. Be absolutely certain that any changes you make are to the SSD and not to some other disk (changes made are irrevocable).

If it turns out that this is shown as the first SATA disk (try “lsblk -f”), then the first partition of the first disk will be “/dev/sda1” (on a host PC this is probably not “sda” since “sda” is more likely part of the host PC). If and only if this is “/dev/sda1” on the Jetson:

sudo mount /dev/sda1 /usr/local
# Or to remove:
sudo umount /usr/local

To mount on some temporary location, e.g., temporary use of “/mnt” as the location, followed by a copy:

sudo mount /dev/sda1 /mnt
cd /usr/local
sudo cp -adpR * /mnt
sudo umount /mnt
sudo mount /dev/sda1 /usr/local
# Now verify the new disk has the right content, and verify it shows up via:
df -H -t ext4

If this works for you, then information can be added on how to mount the partition automatically on each boot. Putting everything on the SSD is a far different level of effort.

Thanks linuxdev. I selectively copied some directories to the SSD and that helped get the software I needed to the board.

I think it’s possible to copy everything but /boot and maybe /etc if you really wanted. I have /usr, /var/, /tmp and /home on an nvme ssd. Most programs will install somewhere under /usr. Overall system performance is much, much, faster now.

Hi mdegans, thanks. I have tried copying the complete /usr contents but it messes up the permissions and I was unable to use even sudo. Did you do anything to change the permissions before mounting the SSD?

How is your ssd formatted? If the partitions are ext4 (or otherwise support unix permissions), linuxdev’s command to copy them seems correct, however I used “rsync -aH …” to copy them by following these instructions instead (note that the commands in that link should be prefixed by “sudo”). Also, in /etc/fstab, I am using the UUID rather than the device name to reference the partition.

“rsync” is actually the preferred tool since it understands special file types. Ordinary recursive copy with permission preservation is ok for “/usr/local/”, but might need to be improved for some of the other “specialty” partitions/directories. If the partition destination is not a Linux filesystem type though, all bets are off…there is no possibility in that case to copy correctly.

I have just tried ext4 partitions for now. No non-linux file systems have been attempted. I tried using rsync and for some reason, even for /usr/local, I was unable to restart. It just got stuck on startup, showed the mouse cursor, showed the debug screen and the mouse cursor again. Basically kept repeating that. I tried by just copying all /usr/local contents to the SSD from a nautilus window and that actually worked fine. I don’t know what I did wrong.

Huh. That shouldn’t actually have worked. Can you do a ls -l within /usr/local ? I’m curious to see the ownership.

Sure. This is the output.

drwxrwxrwx 2 root root 4096 Feb 20 13:59 bin
lrwxrwxrwx 1 root root 9 Feb 20 09:12 cuda -> cuda-10.0
drwxrwxrwx 12 root root 4096 Feb 19 21:36 cuda-10.0
drwxrwxrwx 2 root root 4096 Apr 26 2018 etc
drwxrwxrwx 2 root root 4096 Apr 26 2018 games
drwxrwxrwx 5 root root 4096 Feb 20 13:59 include
drwxrwxrwx 6 root root 20480 Feb 20 13:59 lib
drwxr-xr-x 2 root root 16384 Feb 20 08:56 lost+found
lrwxrwxrwx 1 root root 9 Feb 20 09:12 man -> share/man
drwxrwxrwx 2 root root 4096 Apr 26 2018 sbin
drwxrwxrwx 11 root root 4096 Feb 20 13:59 share
drwxrwxrwx 2 root root 4096 Apr 26 2018 src

Somehow your permissions got mangled. Can you do a “mount | grep usr”. I am curious to see the mount arguments.

Yup. It is

/dev/nvme0n1p5 on /usr/local type ext4 (rw,relatime,data=ordered)

Well. That’s the correct filesystem type all right and mount options look fine, but it looks like copying changed all permissions to 777 somehow. You might attempt to fix it with “chmod -R og-w /usr/local”. /usr/local should look like this afterwards:

$ ls -la
total 60
drwxr-xr-x 11 root root  4096 Jan  7 12:50 .
drwxr-xr-x 11 root root  4096 Jan 31 08:11 ..
drwxr-xr-x  2 root root  4096 Feb  2 22:46 bin
lrwxrwxrwx  1 root root     9 Jan  7 12:50 cuda -> cuda-10.0
drwxr-xr-x 12 root root  4096 Jan  7 12:50 cuda-10.0
drwxr-xr-x  2 root root  4096 Apr 26  2018 etc
drwxr-xr-x  2 root root  4096 Apr 26  2018 games
drwxr-xr-x  3 root root  4096 Jan  8 17:33 include
drwxr-xr-x  7 root root 20480 Feb 19 17:24 lib
lrwxrwxrwx  1 root root     9 Apr 26  2018 man -> share/man
drwxr-xr-x  2 root root  4096 Apr 26  2018 sbin
drwxr-xr-x 10 root root  4096 Jan 31 12:21 share
drwxr-xr-x  2 root root  4096 Apr 26  2018 src

If it fails, you can always unmount /usr/local and the original files should still be there, provided you didn’t move them. You say rsync failed. Did it give you an error message or something?

Thanks for the method to fix the permissions. I am going to hold off on that. I just finished setting up a lot of software I need and its working fine so I am going to hold off on changing anything for now as long as I don’t get any problems.
Rsync didn’t fail but when I mount the hard drive, it doesn’t boot up properly and is stuck in a loop trying to bring up the login screen.

What is the entry in “/etc/fstab” for that mount?

The mounting entry is,
UUID= /home/user/ros_ws ext4 defaults 0 6

A “defaults” for options means the mount will block until complete. If anything is incorrect in the mount entry, and the mount cannot complete, then boot will be stuck there. In the man page for fstab you will find “defaults” as equivalent to this comma delimited entry (if you changed “defaults” to this, then it is the same as doing nothing):


If you were to replace “defaults” with this, then a mount could be tested manually, and if the mount works, then set back to “defaults” (and I assume that “” is an actually partition UID):

UUID=<UUID of HD> /home/user/ros_ws ext4 rw,suid,dev,exec,noauto,nouser,async 0 6

What this does is allow the mount by a command which does not name the disk. These commands should succeed if the UUID and other information works:

sudo mount /home/user/ros_ws
ls /home/user/ros_ws
sudo umount /home/user/ros_ws

Once that works, then you can set for auto mount, but you might also want to make it possible to boot if the partition mount fails. To set this back to auto, and to allow failure without boot halting:

UUID=<UUID of HD> /home/user/ros_ws ext4 rw,suid,dev,exec,auto,nouser,async,nofail 0 6

Thank you for the tons of information. I’ll give it a try after some testing that is currently ongoing on the device.

I put /usr/local /opt and my home directory on the nvme drive. That leaves plenty of space on the internal drive.