Lots of good info in this thread, gtj. Thanks for posting!
Let me just talk about XNX and USB first. Both Nano and NX have a RealTek USB hub on-board (that’s the “USB Device 0bda:5489” you see in during CBoot’s USB boot attempt). Unfortunately, neither CBoot nor U-Boot currently support hubs, so you can’t boot from USB on either platform. We’re working on it, and hope to have USB hub support out in a (near) future release. Note that the kernel DOES support the hub, so you can put data and/or your rootfs there as you’ve stated, just not boot directly from it (still have to boot from SD-card or TFTP).
The XUSB firmware is loaded by CBoot on NX, but not until you attempt to boot from USB. You should see something like “I> USB Firmware Version: 60.06 release” in your XNX log, just prior to the hub enumeration. Nano also loads the XUSB FW in CBoot, but until I can get an XHCI driver loaded in U-Boot, it does nothing.
Note that AGX (the older Xavier platform) has no hub, only 1 USB port on the baseboard. So it is able to boot a USB mass-storage device (stick, drive, etc.), as it always has, as long as there’s no hub in between.
PCIe/NVMe - we’re also working on getting a NVMe driver up for XNX, and we’re updating U-Boot to a recent upstream release (v2020.04) to bring in NVMe support on Nano, too. So you should be able to boot from NVMe in a future release on both Nano and XNX.
Finally, I’m not a kernel dev, but IIRC the kernel takes the ‘root’ args from the LAST one in the command line, i.e. if you have ‘root=/dev/mmcblk0p1’ at the beginning of the cmdline, and ‘root=nfs’, etc. at the end, it will look for the rootfs on NFS. By using the APPEND line in extlinux.conf, you will place your new root= args at the end of the command line, and those should be the ones the kernel honors .