How to install Linux on system disk and CUDA on SSD

We use Jetson TX2 NX and try to install Linux on system disk and CUDA on SSD. How to achieve this?

Most of the CUDA content is in “/usr/local”. See what subdirectories you see from:
ls -d /usr/local/*

Typically, if you have something like an SSD, you can partition it with filesystem type ext4. Then, let’s pretend the partition shows up as “/dev/sda1” (or perhaps “/dev/nvme0n1p1” if it is an NVMe, but I’ll stick to sda1). Assuming you already have some CUDA content on “/usr/local”, you can then mount the drive somewhere temporary to perform copy. sudo umount the partition if it is already mounted, and then “sudo mount /dev/sda1 /mnt” ("/mnt" is a traditional temp mounting point for the system). You could then use any tool to copy content from “/usr/local/*” to “/mnt”.

If we were dealing with special files I’d be tempted to use rsync like this (when not testing and you want to really run an rsync operation remove the “--dry-run” option):

cd /mnt
sudo rsync --dry-run -avczrx /usr/local/* .

You could use that even on device special files, symbolic links, so on. However, because there are no special files in “/usr/local/*” (there are special files in the subdirectories, but we are copying directories), but in this case you could just perform a recursive cp:

cd /mnt
sudo cp /usr/local/* .

The content of the SSD partition at “/mnt” will now contain the content which is in “/usr/local”. If you were to remount this partition to “/usr/local”, then the original content on eMMC or SD card would disappear temporarily until the SSD is umount'd. If you test this mount and it seems good, then you could recursively delete all of the “/usr/local/*” content and just mount the SSD from then on. However, I’d recommend leaving the non-cuda content alone, e.g., leave “/usr/local/bin” alone in case you want to run without the SSD (you’d be able to boot without the SSD and still have a normal system which only lacks the “/usr/local/cuda*”).

Then make a mount rule which tries to mount the SSD, but does not prevent boot if the SSD is missing, and add that rule to “/etc/fstab”. In what follows you can name an exact partition for that exact drive, or you could name it generically as “/dev/sda1”. This first version of such a mount rule using sda1 is (you can add more spaces or tabs between the non-comma-delimited tokens):

/dev/sda1  /usr/local ext4   rw,suid,dev,exec,nofail     0 1

From then on any “/dev/sda1” found would mount there. Since you copied all content from the original “/usr/local” you can expect this to look exactly like what was already there. If the partition is not available, then you’d still have everything except the “cuda*” since you only deleted “cuda*” from the original “/usr/local/*”.

If you want only this exact partition on this exact SSD to mount, then you could use the UUID to perform the identification. Let’s say your SSD is on the system, and that you check it’s ID:
lsblk -f /dev/sda1
…and there is some output similar to this for the partition you have the content mirrored to:

sda
└─sda1      ext4       abcd-1234-ef5678-2468   /usr/local

…then you’d change the fstab entry to this:

UUID=abcd-1234-ef5678-2468  /usr/local ext4   rw,suid,dev,exec,nofail     0 1

…and then the mount rule would work only for that exact SSD partition.

Any “apt-get” or other package command which produces content in “/usr/local” would go to the SSD. Any cuda or bin content run from “/usr/local” would also run from the SSD.