NX DevKit Boot Process: First Impressions

One thing I wanted to mention is that I see no indication of the bootloaders loading the xusb firmware. I would have though I’d have seen some mention of the “xusb-fw” partition being read. Could that be why USB isn’t working?

I have been trying to set the rootfs to the NVMe SSD by modifying the APPEND line:

APPEND ${cbootargs} root=UUID=actual_uuid rw rootwait rootfstype=ext4

However, I would guess that the NVMe disk needs to be mounted during boot at a minimum for this to work. I tried modifying /etc/fstab but was unable to guess the magic spell needed, as it still boots from the SD card.

I made a backup of the entry and renamed it for the sd card, primary points to the modified entry in the extlinux.conf file.

What be the secret?

Well, it’s always going to load the kernel and initrd from the sd card. You can’t change that. The initrd though should be reading the root= and attempting to mount that as /. Well before that, the PCIe bus and NVMe should have been initialized. fstab should not be a factor as it uses /dev/root as a placeholder for the root partition.

Look for lines in your boot log like…

[    4.545222] nvme nvme0: pci function 0005:01:00.0
...
[    4.545256] nvme 0005:01:00.0: enabling device (0000 -> 0002)
...
[    4.656730]  nvme0n1: p1 p2 p3

and then

[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Basic System.
[  OK  ] Found device Samsung SSD 950 PRO 256GB _nvme_root.

Of course, I have a Samsung drive so you may see something else.

Also, do an lsblk to see exactly what was mounted.

I’m guessing that I must have bungled the SSD/partition formatting.
I see the nvme part:
[ 5.424967] nvme nvme0: pci function 0005:01:00.0
[ 5.425027] nvme 0005:01:00.0: enabling device (0000 → 0002)

[ 5.537194] nvme0n1: p1

but the “and then” never happens.
There is no “Found device”, or other statements like that.

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop1          7:1    0    16M  1 loop 
mtdblock0     31:0    0    32M  0 disk 
mmcblk0      179:0    0  59.6G  0 disk 
├─mmcblk0p1  179:1    0  59.1G  0 part /
├─mmcblk0p2  179:2    0    64M  0 part 
├─mmcblk0p3  179:3    0    64M  0 part 
├─mmcblk0p4  179:4    0   448K  0 part 
├─mmcblk0p5  179:5    0   448K  0 part 
├─mmcblk0p6  179:6    0    63M  0 part 
├─mmcblk0p7  179:7    0   512K  0 part 
├─mmcblk0p8  179:8    0   256K  0 part 
├─mmcblk0p9  179:9    0   256K  0 part 
└─mmcblk0p10 179:10   0   300M  0 part 
zram0        252:0    0   1.9G  0 disk [SWAP]
zram1        252:1    0   1.9G  0 disk [SWAP]
nvme0n1      259:0    0 465.8G  0 disk 
└─nvme0n1p1  259:1    0   461G  0 part 

So it does not look like the SSD is mounted, though I can see its shifty little eyes peaking out.

I noticed on your machine it appears to find 3 partitions

[ 4.656730] nvme0n1: p1 p2 p3

How did you set your disk up? I appreciate your help on this.

That’s really weird. How about doing a cat /proc/cmdline and a blkid to verify that the UUIDs match?

I wonder if you need to regenerate your initrd? I don’t recall having to do that but it may help. You’ll have to install the initramfs-tools package first.

  • sudo apt-get install initramfs-tools
  • Update /etc/fstab to use UUID= actual_uuid as the device for /.
  • Make a backup copy of /boot/initrd.img-4.9.140-tegra in case you need to restore it quickly.
  • Run sudo update-initramfs -u

On my nvme, there’s the root filesystem plus a swap partition and a test partition.

Yes it is very strange. Still no joy. Even after modifying the /etc/fstab and regenerating initramfs, it doesn’t mount as root. Is there some magic on how you formatted and partitioned your disk?

$ cat /proc/cmdline
root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0696000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=6 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1 root=UUID=ade44c34-82d5-4043-9112-6c8b2a7e9a47 rw rootwait rootfstype=ext4

$ blkid
...
/dev/nvme0n1p1: LABEL="XavierNX500" UUID="ade44c34-82d5-4043-9112-6c8b2a7e9a47" TYPE="ext4" PARTLABEL="APP" PARTUUID="154b5f72-5c11-4031-9028-e93964a907e4"
...

As suggested, here’s the new /etc/fstab

# <file system> <mount point>             <type>          <options>                               <dump> <pass>
UUID=ade44c34-82d5-4043-9112-6c8b2a7e9a47            /                     ext4           defaults                                     0 1

then I updated the initramfs using the command line you gave, the nvme did not appear to mount:

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0          7:0    0    16M  1 loop 
mtdblock0     31:0    0    32M  0 disk 
mmcblk0      179:0    0  59.6G  0 disk 
├─mmcblk0p1  179:1    0  59.1G  0 part /
├─mmcblk0p2  179:2    0    64M  0 part 
├─mmcblk0p3  179:3    0    64M  0 part 
├─mmcblk0p4  179:4    0   448K  0 part 
├─mmcblk0p5  179:5    0   448K  0 part 
├─mmcblk0p6  179:6    0    63M  0 part 
├─mmcblk0p7  179:7    0   512K  0 part 
├─mmcblk0p8  179:8    0   256K  0 part 
├─mmcblk0p9  179:9    0   256K  0 part 
└─mmcblk0p10 179:10   0   300M  0 part 
zram0        252:0    0   1.9G  0 disk [SWAP]
zram1        252:1    0   1.9G  0 disk [SWAP]
nvme0n1      259:0    0 465.8G  0 disk 
└─nvme0n1p1  259:1    0   461G  0 part

I did a standard gpt table and added 3 partitions with parted. The only thing different is that the partition label on mine is “_nvme_root” instead of “APP”. Maybe having APP on both the sdcard and the nvme is confusing something. I’m grasping at straws.

I’ve played with it a lot, but was unable to make it obey. I ended up using the method here: How to Boot from NVMe SSD? - #22 by crazy_yorick which uses a systemd service to mount the NVMe SSD and switch the rootfs.

The extlinux.conf version still sounds intriguing though.

Very strange but I’m glad you got it working.

1 Like

I’ve used pointed above by Kangalow hack with systemd service because couldn’t make AGX Xavier load rootfs from SSD with “root=…” APPEND in extlinux.conf. I have seen first part of gtj’s log too, but not last part.

gtj, you have very interesting log post:

[ OK ] Reached target System Initialization.
[ OK ] Reached target Basic System.
[ OK ] Found device Samsung SSD 950 PRO 256GB _nvme_root.

It’s systemd log. I don’t have Xavier NX, but Xavier AGX flashed by JetPack 4.4 DP has old-style initrd (file /boot/initrd on mmcblk0p1 ) with init script, without systemd.

Have you remake initrd or in Xavier NX different initrd is used?

I apologize. I use Dracut to generate the initramfs and it prints different messages even though it still starts systemd. I just tried generating an initrd with update-initramfs and while it didn’t generate those messages, it still found the rootfs on the nvme.

Hi,
I have found why APPEND didn’t work with initrd flashed by JetPack 4.4 on Xavier AGX. In AGX thread To work, it requires init script modification in initrd.

In case initrd, created by update-initramfs, it works with just APPEND in extlinux.conf, as you reported early. But in case of Xavier AGX, there is error “xusb firmware not found” at initrd step and usb devices don’t work after boot.

Yeah, that’s a known issue with the Nano as well. The default initrd doesn’t load the USB firmware which prevents it from being used as a rootfs. I outlined the fix in this post… Jetson nano r32.2.1 PARTUUID not working while booting from USB - #9 by gtj

Doesn’t help NVMe though.

Lots of good info in this thread, gtj. Thanks for posting!

Let me just talk about XNX and USB first. Both Nano and NX have a RealTek USB hub on-board (that’s the “USB Device 0bda:5489” you see in during CBoot’s USB boot attempt). Unfortunately, neither CBoot nor U-Boot currently support hubs, so you can’t boot from USB on either platform. We’re working on it, and hope to have USB hub support out in a (near) future release. Note that the kernel DOES support the hub, so you can put data and/or your rootfs there as you’ve stated, just not boot directly from it (still have to boot from SD-card or TFTP).

The XUSB firmware is loaded by CBoot on NX, but not until you attempt to boot from USB. You should see something like “I> USB Firmware Version: 60.06 release” in your XNX log, just prior to the hub enumeration. Nano also loads the XUSB FW in CBoot, but until I can get an XHCI driver loaded in U-Boot, it does nothing.

Note that AGX (the older Xavier platform) has no hub, only 1 USB port on the baseboard. So it is able to boot a USB mass-storage device (stick, drive, etc.), as it always has, as long as there’s no hub in between.

PCIe/NVMe - we’re also working on getting a NVMe driver up for XNX, and we’re updating U-Boot to a recent upstream release (v2020.04) to bring in NVMe support on Nano, too. So you should be able to boot from NVMe in a future release on both Nano and XNX.

Finally, I’m not a kernel dev, but IIRC the kernel takes the ‘root’ args from the LAST one in the command line, i.e. if you have ‘root=/dev/mmcblk0p1’ at the beginning of the cmdline, and ‘root=nfs’, etc. at the end, it will look for the rootfs on NFS. By using the APPEND line in extlinux.conf, you will place your new root= args at the end of the command line, and those should be the ones the kernel honors .

HTH,

Tom

2 Likes

@TWarren Thanks for the update Tom!! It makes perfect sense now why USB boot isn’t working.

You are correct about kernel command line ordering but there’s something fishy with the stock initrd and update-initramfs that’s doesn’t seem to be using either the first or last root= parameter when it tries to pivot to the real root filesystem. I’m still troubleshooting this one though.

Thanks again!

1 Like

Hello @gtj,
I have managed to use update-initramfs generated initrd on Xavier AGX by adding hook to copy xusb firmware: AGX thread . It works with root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX->XXXXXXXXXXXX and for unknown reason doesn’t work with root=/dev/nvme0n1p1. I haven’t tried with root=UUID=actual_uuid APPEND yet.

As for stock initrd from JetPack 4.4 DP, it has very strange commandline parsing in init script ( at AGX thread ):
There is this script (remove .log): init.log (5.7 KB)

62: rootdev="$(sed -ne 's/.*\broot=\/dev\/\([abcdefklmnps0-9]*\)\b.*/\1/p' < /proc/cmdline)"
63: if [ "${rootdev}" == "" ]; then
64: uuid_regex='[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}'
65: rootdev="$(sed -ne "s/.*\broot=\(PARTUUID=${uuid_regex}\)\b.*/\1/p" < /proc/cmdline)"
66:fi

First it search for rootdevice by name “/dev/*” and if it isn’t found, for one set by PARTUUID.
On devices flashed by JetPack first root=/dev/mmcblk0p1 is prepended by cboot to command line, so rootdevice set by

APPEND ${cbootargs} root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX->XXXXXXXXXXXX quiet

will be completely ignored. On line 62 rootdev will be set to “mmcblk0p1” from first “root=…” parameter and lines 64-65 will be skipped.
root=UUID=actual_uuid form is not parsed at all.

So, to change rootfs with APPEND in extlinux.conf now we need to patch init script in stock initrd or generate new initrd.
Or use systemd service method: https://github.com/jetsonhacks/rootOnNVMe

Yeah that makes sense but is very strange. I knew something was messing with that parsing but just didn’t have time to check.

What happens if you add the rootfs parameter before ${cbootargs} on the APPEND line?

PCIe/NVMe - we’re also working on getting a NVMe driver up for XNX, and we’re updating U-Boot to a recent upstream release (v2020.04) to bring in NVMe support on Nano, too. So you should be able to boot from NVMe in a future release on both Nano and XNX.

Are there any plans for AGX nvme boot? I copied my /usr /home /var/lib/docker/ and /tmp over to an ssd and it was working for a few months but this latest update left my system in an unbootable state with:

bash: cannot set terminal process group (-1): Inappropriate ioctl for device

Official, tested, support for this would be very nice. A SD card or even emmc is very very slow in comparison.

Edit: the error seems unrelated to NVMe and my system boots now.

You can always put just the devtree, kernel and initrd to APP partition of sdcard and have rootfs on nvme. That’s what I’m doing and it works fine (initrd needs a small patch to honor rootwait for nvme).

1 Like