It’s very strange that usb firmware is needed to get to an nvme device but if that works, great!
USB firmware is needed because otherwise the usb port will not work.
If you use initrd generated without firmware hook, system do will boot, you will see login screen, but without keyboard it’s useless :).
P.S. I will try APPEND parameter combinations later on weekends.
Yeah but what’s that got to do with nvme unless you’re using a usb<>nvme bridge. I may have gotten lost between this thread and the one on the NX forum. Are you using usb<>nvme or pcie<>nvme?
I use NVMe in M.2 key M slot (PCIe x4).
All above was to differentiate 2 variants of APPEND in extlinux.conf method (2)
2.1 With initrd, installed while flashing with JetPack 4.4.
This case requares patching init script in initrd, as default init script prioritirizes root=/dev/[somedevice] over root=PARTUUID=.... .
With default init APPEND below doesn’t work for some still unknown reason. (Boot stops at initrd’s console, as if there is no device, but I see /dev/nvme0n1p1 there ???)
…
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/nvme0n1p1
In case
INITRD /boot/initrd
APPEND ${cbootargs} root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
first root=/dev/mmcblk0p1 is prioritirized and system boots to rootfs on eMMC.
2.2 With initrd generated by update-initramfs in ubuntu.
In this case Xavier is flashed with JetPack 4.4, then new initrd is generated on Xavier itself and extlinux.conf edited to point on new initrd and APPEND root=PARTUUID= added.
…
INITRD /boot/initrd.img-4.9.140-tegra
APPEND ${cbootargs} root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
This initrd lacks xusb firmware, but has no problems with parsing root=xxx parameter. So linux boots to rootfs on NVMe well, but USB ports are not functional, so no mouse or keyboard could be used.
To add xusb firmware, hook is registered (this post) before update-initramfs.
Thus 2.2 method is most traditional from Linux view, IMHO.
Ah OK, that makes sense. The usb firmware wasn’t needed to make the nvme work, but to make the mouse and keyboard work. Sorry. I was getting confused…
My method,use sdkmanager
on jetson xavier
init nvme ssd and read partuuid
sudo parted /dev/nvme0n1
mklabel gpt
mkpart primary 2048s 100%
quit
sudo mkfs.ext4 /dev/nvme0n1p1
sudo blkid /dev/nvme0n1p1
example my device partuuid
/dev/nvme0n1p1: UUID=“b11f5460-ec67-4f4a-9c41-8cffd4b11df4” TYPE=“ext4” PARTLABEL=“primary” PARTUUID=“532ead91-c15f-4412-83b5-e5675ff21773”
clone emmc to ssd
sudo dd if=/dev/mmcblk0p1 of=/dev/nvme0n1p1
back to linux host
echo ‘532ead91-c15f-4412-83b5-e5675ff21773’ > bootloader/l4t-rootfs-uuid.txt
sudo ./flash.sh jetson-xavier external
on jetson xavier
resize ssd size
sudo resize2fs /dev/nvme0n1p1
upgrade
sudo mount /dev/mmcblk0p1 /mnt
cp -r /boot/* /mnt/
4 Likes
Thanks, @jocover . I have repeated your method. It works great and doesn’t requires any system modifications, unlike APPEND methods. IMHO, it’s the best method in cases where restoration system on eMMC is not required.
I think it should be noted that after sudo ./flash.sh jetson-xavier external all data on eMMC will be rewrited and APP partition consists only boot folder:
./
├── [4.0K] boot
│ ├── [4.0K] dtb
│ │ ├── [269K] tegra194-p2888-0001-p2822-0000.dtb
│ │ └── [4.0K] tegra194-p2888-0001-p2822-0000.dtb.sig
│ ├── [4.0K] extlinux
│ │ └── [ 733] extlinux.conf
│ ├── [ 33M] Image
│ ├── [5.3M] initrd
│ └── [269K] tegra194-p2888-0001-p2822-0000.dtb
└── [ 16K] lost+found [error opening dir]
4 directories, 6 files
Some remarks about kernel command line and initrd.
In @jocover case:
Kernel command line: root=PARTUUID=[MY-PARTUUID] rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0697000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix= boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1 quiet
First root= is from cboot. Now it’s allready in root=PARTUUID= format and for original initrd APPEND ${cbootargs} root=/dev/mmcblk0p1 in extlinux.conf should enable to load rootfs from eMMC (that must be copied there manualy).
About original initrd. It’s main feature - it doesn’t contain any modules, only libs and firmware, so it works with modified kernels ( original dmesg.orig.log (75.6 KB) , build on Xavier (added KVM hw support like there) dmesg.orig-exp2.log (75.9 KB) ):
ls-initrd.orig.log (7.6 KB)
./
…
├── [4.0K] lib
│ ├── [4.0K] aarch64-linux-gnu
│ │ ├── [123K] ld-2.23.so
…
│ │ └── [ 18K] libuuid.so.1.3.0
│ ├── [4.0K] firmware
│ │ ├── [122K] tegra18x_xusb_firmware
│ │ └── [122K] tegra19x_xusb_firmware
│ └── [ 28] ld-linux-aarch64.so.1 → aarch64-linux-gnu/ld-2.23.so
├── [4.0K] mnt
…
22 directories, 137 files
I also have tested with initrd generated by update-initramfs (with xusbfw hook).
Original kernel work fine with initrd generated for it.
Builded on Xavier kernel with it’s initrd failed to boot. Cboot failed to load initrd. I will create new topic for this problem later. initrd-ubuntu-exp2-cutecom.log (23.5 KB)
But modified kernel booted with update-initramfs generated initrd without any taunted modules (dmesg.ubuntu-exp2tgr.log (75.8 KB) ).
Main difference of generated initrd - it’s size. ls-initrd.ubuntu.log (39.2 KB) ls-initrd.ubuntu-exp2.log (39.3 KB)
Thank you for sharing your information.
As I did not have a system running Ubuntu, I was unable to use the SDK method to install JetPack on my Jetson AGX Xavier. I flashed the image to a uSD card, and boot from it.
I would prefer to boot from NVMe SSD. Are any changes required to move the rootfs from uSD to SSD?
Any suggestions about what to do with the eMMC?
ADVAthanksNCE
Jim Brenen shows how to Run From NVMe SSD in this YouTube video.
You can download the scripts from Jim’s github repository.
1 Like
Hi everyone,
I have used system installed on SSD with @jocover 's method without any problem for some weeks. But today Ubuntu Update downloaded some updates from nvidia repository ( this and this ), requested to reboot and then Xavier failed to boot system, hanging on initrd step.
Log from debug port: afterupdate-ubuntu-exp2(tgr)-cutecom.log (27.6 KB)
I tried backup kernel (original), but it panic. afterupdate-ubuntu-orig-cutecom.log (56.2 KB)
There was strange thing in log. After update root=/dev/mmcblk0p1 parameter is back!!! Kernel tried to switch rootfs on /dev/mmcblk0p1, that didn’t contain system in @jocover 's method.
[0015.280] I> Linux Cmdline: root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0697000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1
So, device block with cboot device tree on eMMC was rewrited. I think it’s nvidia-l4t-bootloader package doing.
I had to reflash Xavier from JetPack to repair boot. ( sudo ./flash.sh jetson-xavier external)
System on SSD didn’t take any damage and after reboot system loaded with rootfs from SSD. I copied /boot folder from SSD ( updated from repository kernel and new device trees installed there ) to eMMC and rebooted.
Now Xavier works like before.
So, it should be noted, that now Nvidia firmware updates doesn’t take into account if system was flashed with sudo ./flash.sh jetson-xavier external and reflashing from external PC with JetPack will be needed after update.
Systemd method should be resistant against updates as it doesn’t use bootloader features.
In all cases coping from /boot of SSD to /boot on eMMC will be required.
But today Ubuntu Update downloaded some updates from nvidia repository ( this and this ), requested to reboot and then Xavier failed to boot system, hanging on initrd step.
IMO it’s safer to just put commonly used stuff on a ssd and not the entire of /. I’m not sure how much benefit you get from having stuff like /etc on an ssd, since it’s mostly a bunch of text files.
My partition layout looks like:
nvme0n1 259:11 0 931.5G 0 disk
├─nvme0n1p1 259:12 0 119.2G 0 part /tmp
├─nvme0n1p2 259:13 0 323.9G 0 part /home
├─nvme0n1p3 259:14 0 250G 0 part /usr
└─nvme0n1p6 259:15 0 238.4G 0 part /var/lib/docker
Those are just ext4 partitions on a GPT disk, but you may wish to use something fancier like LVM or btrfs with subvolumes for easier resizing.
1 Like
Yes, if you update system, you need to manually copy /boot of SSD to /boot on emmc
upgrade
sudo mount /dev/mmcblk0p1 /mnt
cp -r /boot/* /mnt/boot/
Why this method is not working for Xavier NX, using
sudo ./flash.sh jetson-xavier-nx-devkit mmcblk0p1
My Jetson Xavier NX still boot from microSD instead of nvme ssd following the same approach, I do not know what went wrong?
Hi, @jocover
Sorry for the late answer.
-
Update, mentioned in my previous post ( 4.4 DP), reflashed CBO partition with default root device path
root=/dev/mmcblk0p1. In this case boot folder copy couldn’t help. I don’t know, was it my local error or previous update feature :) -
According current update to JetPack 4.4 - L4T R32.4.3 production release:
In autoupdate window I have choosen “Reboot later” and have manually copied boot folder to eMMC. Xavier have rebooted succesfully and even with customized kernel, that I have used.
Interesting features:
This update regeneratedextlinux.conffile ( /boot/extlinux/extlinux.conf on SSD). It succesfully parsed my manually edited config (was: extlinux.conf.bak.log (883 Bytes) , now: extlinux.conf.log (1.2 KB) )
root=parameter was added to APPEND:
Before update:
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image-exp2
INITRD /boot/initrd.img-4.9.140-tegra
APPEND ${cbootargs} quiet
After update:
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image-exp2
INITRD /boot/initrd.img-4.9.140-tegra
APPEND ${cbootargs} quiet root=PARTUUID=84bce66b-f20c-49f0-a6dd-d2d402a01189 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4
Kernel command line from dmesg:
[ 0.000000] Kernel command line: console=ttyTCU0,115200 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0697000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1 quiet root=PARTUUID=84bce66b-f20c-49f0-a6dd-d2d402a01189 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4
So, in [JetPack 4.4 - L4T R32.4.3 production release] root device path from CBO partition is not used. I think it’s related to Issue 2926568 from 32.4.3 Release Notes . Problem with init script in initrd has been solved by NVIDIA by removing source of first root= parameter :). Update switched system from your method to boot from SSD to APPEND method.
Your method should be tested on latest JetPack. external option could be unsupported now.
Hi, @crazy_yorick ,
Thanks for your sharing, I follow the steps and run the scripts from “GitHub - jetsonhacks/rootOnNVMe: Switch the rootfs to a NVMe SSD on the Jetson Xavier NX and Jetson AGX Xavier” to switch the rootfs to SSD successfully.
Because the SD Card is still needed for booting, I am now trying to simplify the SD Card, remove the files that has nothing to do with booting and make the SD small. I find the fold usr/ fold contains a lot of files. Could you help specify that which files we need keep for booting and the other files can be removed?
Hi, @laowanggeorge ,
Rootfs switching method is implemented with booting from system on eMMC (SDCard in your case) in SSD fail cases in mind. There are units, executing in parallel with setssdroot.service , including udevd, thus, most system folders on SD card, except “home”, are required (systemd method pros and cons).
If you don’t need to keep spare system and use JetPack 4.4 then IMHO it’s better to use extlinux.conf APPEND method.
In that case only “boot” folder required on SD Card APP partition. “Pros and cons” link above is outdated for APPEND method. Latest cboot in JetPack 4.4 (not DP!!) don’t add root=.. parameter from device tree on CBO partition, so problem with two root=.. parameter in cmdline is solved.
After flashing with latest JetPack there are extlinux.conf ( /boot/extlinux/extlinux.conf) on eMMC/SD card. It already contains APPEND parameter, something like this:
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} quiet root=‘some device’ rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4
So, to place rootfs on SSD with APPEND method you can try 2 ways:
-
Simple, but I haven’t tested it on latest JetPack. Follow @jocover’s method. As result, only boot folder on eMMC will be created.
-
Straightforward method:
- Flash device from SDKManager
- Boot device, configure OS.
- Open
extlinux.conf( /boot/extlinux/extlinux.conf) and in APPEND string replace
root='some device'withroot=PARTUUID=your SSD partuid - Copy rootfs from eMMC/SD to SSD
- Reboot device. It should boot to rootfs on SSD.
After OTA update of bootloader (when after ubuntu autoupdate new window with warning appears), you should click “Reboot later”, mount SD card and copy /boot folder from SSD to SD card. If you build custom kernel or edit extlinux.conf always syncronise boot folders on SSD and SD card. Because OTA update parse extlinux.conf on SSD and can fail if this folder have unexpected files, but flashing bootloader-related partitions (CBO and others) on eMMC/SD will succeed and reflashing device will be needed. I have already bricked my Xavier once with just additional device tree file in boot folder on SSD :( .
Regarding @jocover’s method can somebody explain what the procedure is to accomplish this? I understand what each individual cli command does, but I have the faintest idea how to accomplish this with SDK Manager.
Is there a hidden command line interface/shell/ini config file?
Trying to grasp the caveats.
I can’t help with most of this, but understand that SDK Manager is just a front end for downloading software, running the flash software (as a backend), and other dependency resolving. If you’ve used SDKM to flash once, then you will have this directory:
~/nvidia/nvidia_sdk/JetPack...version.../Linux_for_Tegra/
The “Linux_for_Tegra/” content is what you could have downloaded separately as the “driver package”. This is the true flash software. You don’t even need SDKM to download and unpack this, SDKM is just a convenience for this.
Within “Linux_for_Tegra/” there is an initially blank subdirectory, “Linux_for_Tegra/rootfs/”. When flashing, it is this content which makes up the root filesystem. The downloadable package for “sample root filesystem” is normally unpacked there (using sudo), and is an entirely 100% pure Ubuntu. This lacks some of the needed drivers for Jetsons, and so from “Linux_for_Tegra/”, running command “sudo ./apply_binaries.sh” adds those drivers. This allows NVIDIA’s content to be maintained separately and not be mixed with Ubuntu except when the end user chooses to do so.
The “flash.sh” command is normally how one would flash on command line, and this is in the “Linux_for_Tegra/” directory which is installed when unpacking the driver package.
Flashing does not install any of the optional “extras”, e.g., no CUDA or samples. This is the part where SDKM helps as it can do this for you without many of the steps you might need to understand to add these packages yourself. This includes optional packages to both host PC and Jetson.
Most of what the jocover method does is simply from the host side without involving SDKM other than having the “Linux_for_Tegra/” content. When he gets to this line it is the first time SDKM is involved, and then only for the purpose of downloading the “Linux_for_Tegra/” content:
sudo ./flash.sh jetson-xavier external
Do you have “Linux_for_Tegra/”? If so, then you have no more requirements (within his post) for using SDKM. If you don’t have “Linux_for_Tegra/”, then you could skip SDKM and separately download the driver package and sample rootfs, prepare with a single “sudo apply_binaries.sh”, and from then on run as many flashes as you want without ever touching SDKM.
Note that during a flash the “Linux_for_Tegra/rootfs/” content is almost exactly what is used to create the root filesystem on the Jetson. The exception to being an exact copy is from some of the “/rootfs/boot” content being edited prior to flash, where the edits depend on the argument passed to flash.sh. In this example the argument is “external”, whereas normally it would be “mmblk0p1”. Part of the changes from this argument change is that the “rootfs/boot/extlinux/extlinux.conf” will be edited to point at your partition’s UUID instead of pointing at “/dev/mmcblk0p1” during boot.
Special instructions are almost always referring to something within the “Linux_for_Tegra/” directory. The other commands could be run from anywhere on the host PC for the first part of his instructions. The rest of the commands are from the running Jetson after flash completes and you’ve performed the first boot setup (it is hard to set up if you don’t have a user account, and this is part of first boot setup).
2 Likes