Boot chains differ when using an NVMe. On a system with a BIOS this wouldn’t matter so much, but Jetsons do not have a BIOS. Jetsons do still have that functionality, but it is in software (for eMMC models it tends to be in eMMC partitions; for SD card models, this would likely be in QSPI memory on the module). That “equivalent to BIOS” software is not as flexible, although once it gets to the UEFI stage which is in JP6/L4T R36.x, more uniformity is possible. However, arrangement of things like drivers for the NVMe might or might not still be an issue (somehow the kernel must be loaded, and the boot configuration to pass to the kernel; with multiple devices and possibly transferring control from one device to the next things might not work the way it is expected for an actual BIOS). More on this topic further down.
You did not mention installing all modules, so you might mention if you installed all of the modules.
I should explain a bit about initial ramdisks (an initrd; the INITRD key/value pair in extlinux.conf). Traditional older systems always read the kernel from some “/boot” partition using a filesystem type which was understood by both the bootloader and the kernel. How could you load a kernel on a filesystem if the bootloader itself does not have the knowledge of the filesystem to decode the bytes? However, one might have a working “/boot”, and the rest of the “/” (rootfs) filesystem could be on some filesystem type that the kernel itself understands, and since the bootloader has more or less ended its own life by overwriting itself with the Linux kernel, it no longer matters that the bootloader cannot read this new “mystery” filesystem type. The kernel can do that.
However, kernels did not always have dynamically loadable modules for drivers (trivia: back in my days with the dinosaurs I was using kernel 1.0.13(?memory hazy?), which was monolithic; a short time later the experimental 1.1.59, or something like that, was introduced, and it was the first kernel with loadable modules). This means that the bootloader knew nothing about loadable modules, nor did the bootloader have any design to help this. At this time any driver for the rootfs itself had to be integrated into the kernel and not in the form of a module. “Why would this be mandatory” would be a good question; the reason is that modules themselves are stored within “/lib/modules/”, and so you needed the module to read the rootfs, but the module to read the rootfs was unreachable until you had the module loaded (the proverbial “Catch 22”). Along came the initrd.
If you take a block of RAM, and you cover it with loopback so it looks like a simple filesystem which the bootloader understands, and then copy a tiny and temporary filesystem into it (including a mini init and the kernel modules needed to read the actual hard drive), then you can load the kernel and the module used for whatever odd filesystem type is on the actual hard drive (or something else difficult to configure, e.g., a RAID array). You’re still stuck with a miniature “mini me” Linux, but then a “pivot root” is performed; the kernel has been accessing “/” on a simple ramdisk (initrd since it is the first or “initial” ramdisk used in init), and when the root is mutated to say that it is now the actual hard drive, magic happens and the kernel uses that hard drive as if it was always using it.
There is a very important point here: Any driver needed to run that alternate hard drive must have the driver added to the initrd, otherwise one cannot find the drivers on the hard drive. Any module not mandatory for reading that filesystem does not need to be on the initrd. If you have made a kernel modification which is required for boot, and if any of that is in the form of a module, then it is possible the boot failure is due to missing the driver which runs in initrd prior to pivot root.
This gets a bit more confusing on Jetsons. The explanation will differ between the Orin Nano variants. If you have a dev kit, then the Orin Nano will not have eMMC, but it will have QSPI memory. If you have a commercial Nano (which is normally what is sold with third party commercial carrier boards), then there is eMMC memory. That boot chain which loads the kernel is quite different between those two models. Which Orin Nano model do you have? Is this a commercial module with eMMC on a third party carrier board? Or is this an actual developer’s kit which would normally use only an SD card? Answers change depending on the model.
When adding drivers, sometimes there are options which you may not personally encounter, but which the flash software might try to perform depending on the type of module detected (eMMC model versus SD card plus QSPI model). My thought is that perhaps (I don’t know for certain) the inability to find the filesystem is one of these reasons:
- Some sort of flash changed the partition UUID. Only occurs when flashing creates a partition.
- There is now a missing driver and the device with that PARTUUID requiring the driver to see the partition (e.g., a module was used to understand the partition and filesystem, but it is missing from the
initrd).
- The content being used (drivers and configuration) may not be where you expect it to be, e.g., it is in QSPI of a non-eMMC model, or it is in a partition of an eMMC model), and you’ve updated part of what is needed, but the chain of searching devices has somehow skipped over your configuration and is using some other configuration (keep in mind that booting is a chain of events in multiple memory devices because Jetsons do not have a hardware BIOS; Jetsons have the equivalent of a BIOS in software which might be spread out in ways you don’t know about).
Also, I’m not so certain that gadgetfs is a filesystem type. Certainly “/dev/gadgetfs” would be a block device (e.g., a partition). However, partitions also have a filesystem type, e.g., ext4. I’m not familiar enough with this particular gadget layout (where there is a device special file in /dev; I’ve only worked with loopback covered files directly named in a gadget export). Keep in mind that you can let mount try to auto detect the format. What is your entry in “/etc/fstab” related to this gadget filesystem, if any?
