Add CUDA and other libraries to a custom file system (no gui)

Made a file system with no gui for JETSON NANO, it’s working fine for now, I need to add CUDA and other libraries using sdkmanager, can i flash my current no gui system with those library ? or does it only work for flashing new complete images ?

An explanation of some of how flash sets up a filesystem might help answer this for you, but with some work, the short answer is that you can probably do this.

JetPack/SDKM is not actually the software which flashes. SDKM is a front end for the “driver package”, and it is this package which performs the actual flash (the “driver package” is basically most of the content in the “Linux_for_Tegra/” subdirectory under “~/nvidia/nvidia_sdk/JetPack...version.../Linux_for_Tegra/”).

By default one unpacks the “sample rootfs” into “Linux_for_Tegra/rootfs/”, and this is purely Ubuntu. Licensing is maintained by not mixing NVIDIA content into the Ubuntu content. Then the end user (or JetPack/SDKM acting as a front end) uses the “apply_binaries.sh” script to overlay NVIDIA-specific binaries into the otherwise purely Ubuntu directory tree. This is when it becomes known as “Linux for Tegra” (“L4T”).

In older releases this copy of NVIDIA content was just that…purely a file copy. In more recent releases this content was moved into the Debian package system (the “.deb” files). As such, those files could no longer be copied directly in (sort of like your situation since you are using packages, and not individual files).

Because the “dpkg” tool is needed to install those files, you have to use that tool. The problem is that the native dpkg on the host is x86_64/amd64 architecture, but your Jetson is arm64/aarch64 architecture. The native dpkg won’t work.

So what NVIDIA did during the transition from copying files into rootfs to instead being packages was to use the QEMU emulator. The dpkg within the emulator is a purely arm64/aarch64 environment, but runs in the x86_64/amd64 world. The “apply_binaries.sh” script is what performs the setup and running of the emulator in order to install package content (it also does a chroot to the rootfs/ subdirectory before running).

You could examine how the “Linux_for_Tegra/apply_binaries.sh” script does this, but there is a better example created for a similar purpose. For reasons similar to yours, someone created a script called “l4t_create_default_user.sh”. This script sets up QEMU to pretend it is first boot, and allows the user to set up account name/pass and timezone preferences directly into the “rootfs/” image which was going to be flashed. You could alter that script to install CUDA and other libraries, but beware some of them might trigger install of Xorg content (some of CUDA and GPU drivers are used by loading into the X11 server as a module).

The script can be found here:
https://forums.developer.nvidia.com/t/jetson-nano-all-usb-ports-suddenly-stopped-working/75784/37

If you really wanted to, you could simply build your image up on a running Jetson, e.g., use the “apt-get install ...” method to add your packages, and even update the packages. When you get it where you want, then you’d clone and use the cloned image instead of generating it from scratch.

Incidentally, using your own image from a clone (versus generating it from “rootfs/”) would use command line instead of SDKM. With the Jetson in recovery mode and the USB connected, a typical flash (assuming SD card image…this changes with eMMC) would be:
sudo ./flash.sh jetson-nano-qspi-sd mmcblk0p1

Or for an eMMC model:
sudo ./flash.sh jetson-nano-emmc mmcblk0p1

To reusing any existing rootfs image one would add the “-r” option, and the above become one of:

  • sudo ./flash.sh -r jetson-nano-qspi-sd mmcblk0p1
  • sudo ./flash.sh -r jetson-nano-emmc mmcblk0p1

The “reuse” implies you have a clone. If you’ve used QEMU to modify the “rootfs/”, then you would not use “-r”, and the image would be generated based on “rootfs/” content (which conveniently has had your packages added via QEMU).

There are also other options if your image is from an SD card.

Thanks for taking the time
I used these scripts from someones repo to make the headless image

it boots and all, once in recovery mode the jetson nano no longer outputs hdmi, it is detected by the sdkmanager but promps errors once login and passwd are entered and i press flash

So if I understand, I can just add the cuda and other packages to the rootfs by a jetson nano to match the architecture, before generating the jetson.img ? And would I just install them using apt-get ?

Also the generated user (login,passwd entered in sdkmanager) has no sudo privilages, I’m stuck with no sudo or root account, I’m a little confused about the created user in my image if you could help me get around that, I don’t understand how the only created user has no sudo and I cant login as root using the password

Recovery mode has no HDMI support and will never provide video. This is only for flashing. Login is not possible if in recovery mode. To log in you have to boot normally to Linux.

If your changes are added to the “Linux_for_Tegra/rootfs/” area, then a normal flash which generates an image based on that will also propagate into your running system after the flash is completed. There are some exceptions to this if we are talking about “/boot” or firmware content, but packages added through the dpkg mechanism are not normally part of “/boot”, so these are mostly safe. An example of something which might fail to propagate if you put it directly into the “rootfs/” tree of the host PC are kernel changes and device tree changes. Libraries and executables should “just work”.

You could also take an existing image, alter that on loopback (an image is a file which needs to be treated as a partition, and loopback is how you do that), and then flash without generating a new image (the “-r” option prevents “Linux_for_Tegra/bootloader/system.img” from being overwritten, it is actually system.img which becomes the partition).

To alter an image on the PC would imply loopback mounting the image, and then using QEMU to run your package operations. The alternate, if you are changing the “rootfs/” content, is the run QEMU in the same way on that subdirectory (versus a loopback mounted subdirectory).

The advantage of using “-r” on an image is that even the “/boot” content is preserved for an image, including kernel and device tree customizations.

The generated user does have sudo privileges in the default system. One of two things might be causing the failed ability to use sudo:

  • Your user was not created with the admin ability, and is not in “admin” group. The default NVIDIA uses is to add the user to supplemental group “admin”. The script from the URL I gave does add to supplemental group “admin”.
  • Your image might lack supplemental group “admin”, which would be equivalent to not adding the user to “admin”, but the cause would be different.
  • Your sudo binary itself was not installed correctly. Note that the binary has to have the correct SUID permission bit, and that if at any point you did not use root/sudo to create this, or if at any point this was copied to a non-Linux native filesystem type, then you would have lost the SUID bit. The output of “ls -l which sudo" should look like this: -rwsr-xr-x. 1 root root 136616 Jan 31 2020 /usr/bin/sudo* ...where the "s`” is the SUID bit.

If you have your image on the PC you could use ls -l on the sudo file and see what its permissions are. You could also look at the “/etc/group” file and see if “admin” exists. If it does exist, then you could see if your user is listed as a member of that group.

/user/bin/sudo has this -rwxr-xr-x 1 root root

and cat /etc/group | grep admin outputs nothing

thanks for the explaination about altering the image will be trying that once the user thing is fixed

but my user is in the sudo group

The user sudo problem is solved, all I did was mount sd card to pc and chmod u+s to /usr/bin/sudo

The flash was improperly completed for the above to occur. No SUID bit. A typical example of this occurring is if the content were unpacked without sudo on the host PC (not an issue if you used JetPack/SDKM to flash, but it can be if you manually unpacked the rootfs), or else the content was copied to a non-Linux partition prior to being used (non-Linux filesystems have no understanding of Linux permissions and simply throw those permissions away).

You will need to re-flash. When the sudo command loses its SUID bit, then it means there are a lot of other files which are also going to have incorrect permissions.

Even if you did correct the SUID bit you’d still need to find a way to add your user to supplemental group admin (which would mean the user is an “sudoer”)…which requires a user who is already an admin or else requires actual root login.

You can clone your content (or copy the SD card) prior to starting just so you have a reference. The operating system content is an issue, but if you happen to have some content you worked on, then the clone would still be useful.

Yeah I was worried if this happened to that file, what about others, but so far all is good, if I come into other problems i’ll just redo the whole thing

Note that my user is in the sudo group, no admin group in /etc/group
Once the chmod is done I could use sudo without problem

Yes, I guess as long as you are in the sudo group this would work. Different releases of even the same distribution tend to change how they arrange account groups, it is a bit of whatever philosophy a given distribution takes for a given release as to how groups are arranged. I just realized that the Ubuntu 18.04 systems don’t even have group “admin”, but instead have “lpadmin” for printers…plus the sudo group. I just transferred my own system from Fedora, so I am kind of mixing them up a bit.

The same advice though: When permissions were wrong like this, then you are better off flashing again and reinstalling because the number of permission issues will be too many.

I’ll be doing that soon, Thank you very much for taking the time