NX DevKit Boot Process: First Impressions

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

If you’re coming from the Nano platform, there’s good news and bad news.
Mostly good; :)

The good news:

  • No more u-boot. CBoot now retrieves extlinux.conf and takes care of
    displaying a boot menu on the serial console and the loading of
    the kernel initrd/initramfs and fdt (if you have one).

  • All of the boot loaders that used to be in the 10 or 14 partitions
    on the SDCard or eMMC, are in the 32MB flash on the module.
    Although the SDCard image does contain 11 partitions, they are
    for recovery and using the A/B boot slot facility. The kernel
    and DTB partitions on the card are only used if CBoot can’t
    find /boot/extlinux/extlinux.conf or if you don’t have an FDT
    entry in extlinux.conf. I’m running now with only 1 partition
    named “APP” on the SDCard and all that has on it is /boot. This
    should pave the way for supporting other distributions.

  • Although you can’t use NVMe as the location of the boot partition,
    nothing special is needed to get it to work as the root filesystem.
    Just update the APPEND line in extlinux.conf to point ot the
    correct root device.

The bad news:

  • It appears you still need the APP partition on either the SDCard
    or eMMC. There’s no support for PCIe/NVMe in CBoot. Although
    the documentation and the console messages imply that USB storage
    is supported, I can’t seem to find a device that qualifies.
    I’ve tried a USB<>SDCard adapter, a USB<>SATA bridge, and a simple
    USB stick and all fail to be recognized with the following
    error:
Enumerated USB Device 0bda:5489
[0005.207] I>
[0005.208] E> Enumerated device doesn't belong to MSD class or protocol is not bulk-only!!
[0005.216] E> USBMSD: Failed to initialize Enumerated USB device either not an MSD device or supported protoco[0005.226] E> USBMSD: Failed to open usbmsd open, 0.
[0005.230] E> Error opening USBMSD driver 0, err: 7c7c0312
[0005.236] E> Failed to initialize device 5-0
[0005.240] E> USB boot failed, err: 2088502034

Anyway, I’m still exploring but this is a leap ahead!

1 Like
3

Hi gtj,

It appears you still need the APP partition on either the SDCard
or eMMC. There’s no support for PCIe/NVMe in CBoot. Although
the documentation and the console messages imply that USB storage
is supported, I can’t seem to find a device that qualifies.
I’ve tried a USB<>SDCard adapter, a USB<>SATA bridge, and a simple
USB stick and all fail to be recognized with the following
error:

Xavier could support

  1. rootfs over USB storage
  2. kernel/kerenl-dtb/boot.img over usb storage.

Which kind did you hit that error? You mentioned some adapters here. Have you tried a native USB drive?

Also, NX is more near to AGX xavier instead of Nano actually …because they are both t194 SoC while Nano is using T210 SoC as TX1.

4

Rootfs on USB works fine as long as the xusb firmware is loaded in the initramfs. I was just trying to find a way to not need the SDCard for the APP partition. Ideally, I’d like CBoot to have access to PCIe/NVMe devices but I’m not sure if that’s feasible.

Can you give me an example of a “native” USB drive? I would have thought a simple USB thumb drive would be the closest to “native”. To be honest, I don’t really need the capability to load from USB. I was just testing things out.

Yes, hence my comment about “If you’re coming from the Nano platform” :) Folks on the AGX platform already had this capability. I feel that the NX is a great “next step up” from the Nano.

5

One thing I wanted to mention is that I see no indication of the bootloaders loading the xusb firmware. I would have though I’d have seen some mention of the “xusb-fw” partition being read. Could that be why USB isn’t working?

6

I have been trying to set the rootfs to the NVMe SSD by modifying the APPEND line:

APPEND ${cbootargs} root=UUID=actual_uuid rw rootwait rootfstype=ext4

However, I would guess that the NVMe disk needs to be mounted during boot at a minimum for this to work. I tried modifying /etc/fstab but was unable to guess the magic spell needed, as it still boots from the SD card.

I made a backup of the entry and renamed it for the sd card, primary points to the modified entry in the extlinux.conf file.

What be the secret?

7

Well, it’s always going to load the kernel and initrd from the sd card. You can’t change that. The initrd though should be reading the root= and attempting to mount that as /. Well before that, the PCIe bus and NVMe should have been initialized. fstab should not be a factor as it uses /dev/root as a placeholder for the root partition.

Look for lines in your boot log like…

[    4.545222] nvme nvme0: pci function 0005:01:00.0
...
[    4.545256] nvme 0005:01:00.0: enabling device (0000 -> 0002)
...
[    4.656730]  nvme0n1: p1 p2 p3

and then

[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Basic System.
[  OK  ] Found device Samsung SSD 950 PRO 256GB _nvme_root.

Of course, I have a Samsung drive so you may see something else.

Also, do an lsblk to see exactly what was mounted.

8

I’m guessing that I must have bungled the SSD/partition formatting.
I see the nvme part:
[ 5.424967] nvme nvme0: pci function 0005:01:00.0
[ 5.425027] nvme 0005:01:00.0: enabling device (0000 → 0002)
…
[ 5.537194] nvme0n1: p1

but the “and then” never happens.
There is no “Found device”, or other statements like that.

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop1          7:1    0    16M  1 loop 
mtdblock0     31:0    0    32M  0 disk 
mmcblk0      179:0    0  59.6G  0 disk 
├─mmcblk0p1  179:1    0  59.1G  0 part /
├─mmcblk0p2  179:2    0    64M  0 part 
├─mmcblk0p3  179:3    0    64M  0 part 
├─mmcblk0p4  179:4    0   448K  0 part 
├─mmcblk0p5  179:5    0   448K  0 part 
├─mmcblk0p6  179:6    0    63M  0 part 
├─mmcblk0p7  179:7    0   512K  0 part 
├─mmcblk0p8  179:8    0   256K  0 part 
├─mmcblk0p9  179:9    0   256K  0 part 
└─mmcblk0p10 179:10   0   300M  0 part 
zram0        252:0    0   1.9G  0 disk [SWAP]
zram1        252:1    0   1.9G  0 disk [SWAP]
nvme0n1      259:0    0 465.8G  0 disk 
└─nvme0n1p1  259:1    0   461G  0 part

So it does not look like the SSD is mounted, though I can see its shifty little eyes peaking out.

I noticed on your machine it appears to find 3 partitions

[ 4.656730] nvme0n1: p1 p2 p3

How did you set your disk up? I appreciate your help on this.

9

That’s really weird. How about doing a cat /proc/cmdline and a blkid to verify that the UUIDs match?

I wonder if you need to regenerate your initrd? I don’t recall having to do that but it may help. You’ll have to install the initramfs-tools package first.

  • sudo apt-get install initramfs-tools
  • Update /etc/fstab to use UUID= actual_uuid as the device for /.
  • Make a backup copy of /boot/initrd.img-4.9.140-tegra in case you need to restore it quickly.
  • Run sudo update-initramfs -u

On my nvme, there’s the root filesystem plus a swap partition and a test partition.

10

Yes it is very strange. Still no joy. Even after modifying the /etc/fstab and regenerating initramfs, it doesn’t mount as root. Is there some magic on how you formatted and partitioned your disk?

$ cat /proc/cmdline
root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069c000 lut_mem=0x2008@0xa0696000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=6 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1 root=UUID=ade44c34-82d5-4043-9112-6c8b2a7e9a47 rw rootwait rootfstype=ext4

$ blkid
...
/dev/nvme0n1p1: LABEL="XavierNX500" UUID="ade44c34-82d5-4043-9112-6c8b2a7e9a47" TYPE="ext4" PARTLABEL="APP" PARTUUID="154b5f72-5c11-4031-9028-e93964a907e4"
...

As suggested, here’s the new /etc/fstab

# <file system> <mount point>             <type>          <options>                               <dump> <pass>
UUID=ade44c34-82d5-4043-9112-6c8b2a7e9a47            /                     ext4           defaults                                     0 1

then I updated the initramfs using the command line you gave, the nvme did not appear to mount:

$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0          7:0    0    16M  1 loop 
mtdblock0     31:0    0    32M  0 disk 
mmcblk0      179:0    0  59.6G  0 disk 
├─mmcblk0p1  179:1    0  59.1G  0 part /
├─mmcblk0p2  179:2    0    64M  0 part 
├─mmcblk0p3  179:3    0    64M  0 part 
├─mmcblk0p4  179:4    0   448K  0 part 
├─mmcblk0p5  179:5    0   448K  0 part 
├─mmcblk0p6  179:6    0    63M  0 part 
├─mmcblk0p7  179:7    0   512K  0 part 
├─mmcblk0p8  179:8    0   256K  0 part 
├─mmcblk0p9  179:9    0   256K  0 part 
└─mmcblk0p10 179:10   0   300M  0 part 
zram0        252:0    0   1.9G  0 disk [SWAP]
zram1        252:1    0   1.9G  0 disk [SWAP]
nvme0n1      259:0    0 465.8G  0 disk 
└─nvme0n1p1  259:1    0   461G  0 part
11

I did a standard gpt table and added 3 partitions with parted. The only thing different is that the partition label on mine is “_nvme_root” instead of “APP”. Maybe having APP on both the sdcard and the nvme is confusing something. I’m grasping at straws.

12

I’ve played with it a lot, but was unable to make it obey. I ended up using the method here: How to Boot from NVMe SSD? - #22 by crazy_yorick which uses a systemd service to mount the NVMe SSD and switch the rootfs.

The extlinux.conf version still sounds intriguing though.

13

Very strange but I’m glad you got it working.

1 Like
14

I’ve used pointed above by Kangalow hack with systemd service because couldn’t make AGX Xavier load rootfs from SSD with “root=…” APPEND in extlinux.conf. I have seen first part of gtj’s log too, but not last part.

gtj, you have very interesting log post:

[ OK ] Reached target System Initialization.
[ OK ] Reached target Basic System.
[ OK ] Found device Samsung SSD 950 PRO 256GB _nvme_root.

It’s systemd log. I don’t have Xavier NX, but Xavier AGX flashed by JetPack 4.4 DP has old-style initrd (file /boot/initrd on mmcblk0p1 ) with init script, without systemd.

Have you remake initrd or in Xavier NX different initrd is used?

15

I apologize. I use Dracut to generate the initramfs and it prints different messages even though it still starts systemd. I just tried generating an initrd with update-initramfs and while it didn’t generate those messages, it still found the rootfs on the nvme.

16

Hi,
I have found why APPEND didn’t work with initrd flashed by JetPack 4.4 on Xavier AGX. In AGX thread To work, it requires init script modification in initrd.

In case initrd, created by update-initramfs, it works with just APPEND in extlinux.conf, as you reported early. But in case of Xavier AGX, there is error “xusb firmware not found” at initrd step and usb devices don’t work after boot.

17

Yeah, that’s a known issue with the Nano as well. The default initrd doesn’t load the USB firmware which prevents it from being used as a rootfs. I outlined the fix in this post… Jetson nano r32.2.1 PARTUUID not working while booting from USB - #9 by gtj

Doesn’t help NVMe though.

18

Lots of good info in this thread, gtj. Thanks for posting!

Let me just talk about XNX and USB first. Both Nano and NX have a RealTek USB hub on-board (that’s the “USB Device 0bda:5489” you see in during CBoot’s USB boot attempt). Unfortunately, neither CBoot nor U-Boot currently support hubs, so you can’t boot from USB on either platform. We’re working on it, and hope to have USB hub support out in a (near) future release. Note that the kernel DOES support the hub, so you can put data and/or your rootfs there as you’ve stated, just not boot directly from it (still have to boot from SD-card or TFTP).

The XUSB firmware is loaded by CBoot on NX, but not until you attempt to boot from USB. You should see something like “I> USB Firmware Version: 60.06 release” in your XNX log, just prior to the hub enumeration. Nano also loads the XUSB FW in CBoot, but until I can get an XHCI driver loaded in U-Boot, it does nothing.

Note that AGX (the older Xavier platform) has no hub, only 1 USB port on the baseboard. So it is able to boot a USB mass-storage device (stick, drive, etc.), as it always has, as long as there’s no hub in between.

PCIe/NVMe - we’re also working on getting a NVMe driver up for XNX, and we’re updating U-Boot to a recent upstream release (v2020.04) to bring in NVMe support on Nano, too. So you should be able to boot from NVMe in a future release on both Nano and XNX.

Finally, I’m not a kernel dev, but IIRC the kernel takes the ‘root’ args from the LAST one in the command line, i.e. if you have ‘root=/dev/mmcblk0p1’ at the beginning of the cmdline, and ‘root=nfs’, etc. at the end, it will look for the rootfs on NFS. By using the APPEND line in extlinux.conf, you will place your new root= args at the end of the command line, and those should be the ones the kernel honors .

HTH,

Tom

2 Likes
19

@TWarren Thanks for the update Tom!! It makes perfect sense now why USB boot isn’t working.

You are correct about kernel command line ordering but there’s something fishy with the stock initrd and update-initramfs that’s doesn’t seem to be using either the first or last root= parameter when it tries to pivot to the real root filesystem. I’m still troubleshooting this one though.

Thanks again!

1 Like
20

Hello @gtj,
I have managed to use update-initramfs generated initrd on Xavier AGX by adding hook to copy xusb firmware: AGX thread . It works with root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX->XXXXXXXXXXXX and for unknown reason doesn’t work with root=/dev/nvme0n1p1. I haven’t tried with root=UUID=actual_uuid APPEND yet.

As for stock initrd from JetPack 4.4 DP, it has very strange commandline parsing in init script ( at AGX thread ):
There is this script (remove .log): init.log (5.7 KB)

62: rootdev="$(sed -ne 's/.*\broot=\/dev\/\([abcdefklmnps0-9]*\)\b.*/\1/p' < /proc/cmdline)"
63: if [ "${rootdev}" == "" ]; then
64: uuid_regex='[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}'
65: rootdev="$(sed -ne "s/.*\broot=\(PARTUUID=${uuid_regex}\)\b.*/\1/p" < /proc/cmdline)"
66:fi

First it search for rootdevice by name “/dev/*” and if it isn’t found, for one set by PARTUUID.
On devices flashed by JetPack first root=/dev/mmcblk0p1 is prepended by cboot to command line, so rootdevice set by

APPEND ${cbootargs} root=PARTUUID=XXXXXXXX-XXXX-XXXX-XXXX->XXXXXXXXXXXX quiet

will be completely ignored. On line 62 rootdev will be set to “mmcblk0p1” from first “root=…” parameter and lines 64-65 will be skipped.
root=UUID=actual_uuid form is not parsed at all.

So, to change rootfs with APPEND in extlinux.conf now we need to patch init script in stock initrd or generate new initrd.
Or use systemd service method: https://github.com/jetsonhacks/rootOnNVMe

21

Yeah that makes sense but is very strange. I knew something was messing with that parsing but just didn’t have time to check.

What happens if you add the rootfs parameter before ${cbootargs} on the APPEND line?