NFS boot does not work on USB-eth

[jurgis.lingys]

Hello. We are using L4T 32.7.5 for both nano and xavier nx boards and NFS boot is really important for the application.

Right now we are using connectech quark board and our own board that is based on devkit board.
We get into NFS boot by editing extlinux.conf

NFS boot works correctly on from the LAN NFS server for:

Nano+our board, using usb2eth chip.
Nano+connetech quark using PCIE onboard ETH.
Xavier+connetech quark board using PCIE onboard ETH.

NFS boot does not work on:

Xavier+ our board using usb2eth chip.

Procedure goes like:

• Setup NFS rootfs on linux host pc
• edit extlinux.conf with the appropriate settings
• if system is xavier sign the extlinux.conf with tegrasign thingy
• rewrite exlinux.conf in the emmc for the nano/xavier modules, if module is xavier also add signature file
• reboot
• system boots to rootfs over LAN

extlinux.conf APPEND lines that we use:
the $MY_IP and so on are generated during generation of the extiliux.conf files, quite obvious what those are.

nano+ our board

APPEND \${cbootargs} root=/dev/nfs rw netdevwait ip=:::::eth1:on ip=:::::eth1:on nfsroot=$MY_IP:$NFS_RFS_PATH console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1

nano+ connectech quark board

APPEND \${cbootargs} root=/dev/nfs rw netdevwait ip=:::::eth0:on nfsroot=$MY_IP:$NFS_RFS_PATH console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1

xavier+ connectech board

APPEND \${cbootargs} root=/dev/nfs rw netdevwait ip=:::::eth1:on nfsroot=$MY_IP:$NFS_RFS_PATH console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1

xavier + our board that doesnt work

APPEND \${cbootargs} root=/dev/nfs rw netdevwait ip=:::::eth0:on ip=:::::eth1:on nfsroot=$MY_IP:$NFS_RFS_PATH console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1

This should have DHCP on both eth0 and eth1 so the board can easily connect to the LAN NFS, but it doesnt.

I tried booting to the emmc root FS and checked dmesg to deduce the problems. What we do in our distro is we rename ethernet interfaces with sytemD after the boot into emmc, for consistency with hardware layouts. I can clearly see comparing nano dmesg to xavier dmesg when paired with the usb2eth chip that in dmesg from nano the smsc95xx is loaded at 1.6 seconds in the boot, meanwhile on xavier smsc95xx is only loaded by systemd. This begs a question, is it possible that the extlinux environment for xavier does not load smsc95xx driver?
If so how can i make it so the driver is loaded.

For the boards we use flash.sh conf files from nvidia and connectech:
nano+our board:
./flash.sh --no-root-check jetson-nano-emmc mmcblk0p1
nano+connectech quark:
./flash.sh --no-root-check base-nano mmcblk0p1
xavier+connectech:
./flash.sh --no-root-check base-xavier mmcblk0p1
xavier+our board (THIS IS THE ONE THAT DOES NOT SEE smsc95xx in bootloader or is not capable of using it):
./flash.sh $1 --no-root-check jetson-xavier-nx-devkit-emmc mmcblk0p1

The dmesg outputs for the boards after booting into emmc can be seen here, plus i added the serial ouput from xavier+our board when its booting into emmc, im no expert expert in this but it looks like smsc95xx is loaded not in bootloader but in the rootfs, thats why it cant get DHCP and boot into NFS when using the extlinux command i shown above. I hope you understand what our issue is and know how this can be fixed :)

dmesg xavier-e1350-NFSBROKEN.log (60.6 KB)
dmesg-CTI-xavier-nfsCapable.log (58.3 KB)
dmesg-nano-e1350-nfsCapable.log (58.4 KB)
xavier-e1350-serial-NFSBROKEN.log (39.1 KB)

we use this procedure to update full rootFS over eth port, if theres another way to do so, over eth to rewrite full rootFS on l4t 32.7.5 then i am glad to hear out what other way we can do so if there is no solution for the smsc95x. We need to do this over smsc95x because we have deployed systems with the boards where nvidias integrated eth is not accessible from the outside, only smsc95x.

[linuxdev]

I cannot answer this, someone else will need to add comment. However, something which is rather important: systemd only runs once the Linux kernel is up. For an IP address to be added in the boot chain the driver and any other code must be present in the boot content, independent of anything which is part of Linux. Do you have the drivers or other required ethernet content in the boot content, or is it just present only in the rootfs before rootfs is mounted?

Another complication is that if you use an initrd, then the content will have to be in the initrd as well. Sometimes people update a driver and it never gets added to the initrd. This latter only matters if the content is used in boot, but an NFS rootfs would matter in boot. Double check any initrd and double check anything related to NFS or ethernet in the boot content.

[KevinFFF]

Hi jurgis.lingys,

Do you refer to Flashing for NFS as Root to setup for nfs boot?

@jurgis.lingys

xavier+our board (THIS IS THE ONE THAT DOES NOT SEE smsc95xx in bootloader or is not capable of using it):
./flash.sh $1 --no-root-check jetson-xavier-nx-devkit-emmc mmcblk0p1

Your flash command seems not correct.

Nano+our board, using usb2eth chip.
Xavier+ our board using usb2eth chip.

Are you using R32.7.5 for both 2 cases?

Please also share the full serial console in your failed case(Xavier+ our board using usb2eth chip)

[jurgis.lingys]

Hi, thanks for the reply!
I tried mounting official initrd for nano and for xavier, but the filesystem layout is not posix, so i have no idea how to check if the driver is available there. Then greping for .ko or for smsc95x i could not find any leads in neither initrd. Of course i understand that smsc95x needs to be loaded in the bootloader, not systemd, thats why i noted that. you could see e1390-xavier log its possible to see my USB eth device, Bus 001 Device 006: ID 0424:ec00 ( is partially is loaded PRE systemd at around 1.6 secons, but it only gets a name after systemd). This works correct in nano, but not xavier nx, which is the weird part, as both initrd are default ones provided for reference emmc flash for the devkit board.

both systems are on l4t 32.7.5, both use official emmc dtb and physically same motherboard.

@KevinFFF
Hello, thanks for you reply!
Unfortunately we are using a little different procedure to what you described with flash.sh.

What our usual full update looks like:

1. Flash rootfs for emmc with l4t and use it normaly (users use this long term)
2. Because devices are physically not reachable we use following steps to reflash the systems over eth.
3. when rootfs needs a big update and/or we want to completely wipe all user data we start NFS server with minimal root fs on the linux PC and prepare a system.img.
4. Edit extlinux.conf, so the jetson mounts minimalRootFS from the NFS on the LAN.
5. From this mounted NFS minimal rootFS, we take the normal system.img and overwrite mmcblk0p1 with a system.img that is on the NFS server by using “dd if=/mnt/system.img.raw of=/dev/mmcblk0p1 bs=4M status=progress”. At this point jetson is fully booted from NFS, and its overwriting mmcblk from the system.img that is stored on NFS.
6. reboot the system. Now extlinux.conf in mmcblk0p1 is rewritten with the one that was in the system.img and jetson boots to the freshly expanded rootFS on the mmcblk0p1

This procedure works on nano+usb2eth, nano+pcie ethernet, xavier+pcie ethernet, but it never works on xavier+usb2eth. The failing xavier log is already attached, I will attach the correctly working nano log tomorrow and will make new xavier log, because everytime this fails i need to reflash modules and set everything up, takes a ton of time. Please read carefully, i already put alot of research into this issue, and provided a ton of info.

[linuxdev]

Just to clarify, if you are to work with NFS prior to the Linux kernel taking over, then the bare metal boot chain, including bootloader, will mandate that (A) network drivers are in place for any NIC, and that (B) NFS filesystem drivers are also present in the bare metal boot chain. During this very early stage Linux itself does not yet even exist. Once you reach the Linux kernel loading, then it is the kernel which must have ethernet and NFS drivers. An initrd is just a special case of the Linux kernel already being loaded.

To clarify, if the kernel itself is in a partition or other part of the Xavier NX’s memory, then it is possible to load the Linux kernel from that, and no NFS is involved. At that point you would want an initrd to contain any special drivers for that situation unless the drivers are integrated directly in the kernel. For a Jetson, there are reasons why you might want to use an initrd anyway even if the kernel has the NFS and ethernet content directly compiled in (versus as a module).

Think about this: If the kernel Image is on the NFS share, then there is no way to load the kernel unless the boot chain itself understands that NFS share. You’d never get to the initrd or kernel load for that case. However, if the Linux kernel is itself not on the NFS share, but is local to the Jetson, then you can load the Linux kernel and not care what the bootloader knows about NFS; this is because the initrd can have the NFS and ethernet drivers in it (or the kernel can have this directly), and you already have the kernel.

I don’t know if you are trying to load a kernel from NFS, or if the kernel is loaded, and then the kernel is running NFS to mount the rootfs.

As an additional note, since your NIC is USB, then the boot stage which relies on NFS must have not only ethernet drivers and NFS drivers, it must also have the USB drivers. Usually this latter is not an issue, but it illustrates that bootloader drivers have to be complete if bootloader is to be the original NFS source of the kernel.

Note that an initrd is a simple cpio compressed filesystem. When you flash a Jetson that needs an initrd, then you use the initrd flash command instead of the usual flash.

An initrd exists in RAM. If you have the initrd file itself (which is decompressed and deserialized to put it in RAM), then you can observe this to understand it:

# Assuming the file name is "initrd".

# This decompresses, but it is still cpio serialized.
gunzip < initrd > initrd.cpio

# Now deserialize. "sudo" is because content is owned by root:
sudo cpio -vid < initrd.cpio

# Now check for subdirectory content. This will be a bash script.

If you use the correct NVIDIA documentation for your release, then it will probably tell you how to modify content for the initrd which will exist after flash with the initrd flash method. It is incorrect to use any initrd other than the one intended for the Xavier NX on the specific carrier board.

To further illustrate, if you have a kernel module (and kernel modules are not enough if the kernel itself is on NFS), then when it loads it will load at a subdirectory from here:
/lib/modules/$(uname -r)/kernel/

Let’s pretend that your kernel is 4.9.230-tegra when you run the command “uname -r”. This means modules are always searched for at:
/lib/modules/4.9.230-tegra/kernel/

If this is on the filesystem, then that has to be there. If you have a module required during boot, then that module file must also exist at that same location, but within the initrd.

I don’t know at what stage your kernel itself is loaded. I don’t know if it is in the NFS content, or whether it is on the TX2’s memory. That determines whether or not NFS+USB+ethernet drivers need to be (A) in the bootloader, or be in (B), the module directory of the initrd and NFS.

[jurgis.lingys]

Hello, please check the previous reply too, it explain our procedure better. As mentioned above this is the serial logs from nfs booting.
Nano+ official emmc dtb + our board boots to NFS no problem with extlinux.conf:

APPEND ${cbootargs} root=/dev/nfs rw netdevwait ip=:::::eth1:on nfsroot=172.23.3.111:/home/raynoxu/Repos/jetson-bsp2/images/nfs/rootfs console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1

This is serial LOG of working example as per requested:
Nano rootFS working.txt (75.0 KB)

Xavier + official emmc dtb + our board + this extlinux.conf, does not boot into NFS. I suspect smsc95xx driver being loaded later than expected.

APPEND ${cbootargs} root=/dev/nfs rw netdevwait ip=:::::eth0:on ip=:::::eth1:on nfsroot=172.23.1.112:/home/raynoxu/Repos/jetson-bsp2/images/nfs/rootfs console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1

xavier NFS serial not woerk.txt (96.0 KB)

It should be possible to recreate this issue with xavier+devkit +smsc95x based usb adapter dongle.

[jurgis.lingys]

Thank you so much for the reply!
So it basically it works as I expected, thanks for the awesome recap/summary!
I did check the initrd files I mounted them as suggested here:
https://www.linuxquestions.org/questions/linux-newbie-8/command-help-to-mount-initrd-img-and-add-files-to-it-to-create-bootloader-640206/
The inside of inird looks like this. I did search for ko files or find | grep “*.ko” and find | grep “smsc95x” but i did not find anything in the initrd (nothing in nano initrd, nothing in xavier initrd).

Exploring the web i found this post, i will give it a go, i suspect it could be that driver is working but for some reason the DHCP does not want to work when running from BL. Anyways thanks for great help!
https://forums.raspberrypi.com/viewtopic.php?t=107293

[linuxdev]

I have not checked those other URLs, but if no kernel module is required for booting, then none will exist in the initrd. The directory “/lib/modules/$(uname -r)/” should exist in the initrd even if there is no actual kernel module required. Once the initrd is done it will pivot root to the actual rootfs device, and presumably any required modules to continue on are in that device’s /lib/modules/$(uname -r)/. Actual initrd content is very sparse other than perhaps firmware or a module and minimal bash scripts.

Incidentally, a better way to copy and paste results from a directory tree, without using screenshots:

sudo apt-get install tree
cd /where/ever/your/initrd/unpack/is
tree
# Or "tree -d" if you only want to see directories.

DHCP requires the Jetson to request an address. If that request occurs prior to the Linux kernel running, then it means that all drivers must be added to the boot chain (it isn’t a question of Linux seeing this if we speak of time before Linux existing). The boot chain might also need to be modified to perform the DHCP request (drivers alone are not sufficient; the NIC must be told to broadcast that DHCP request). Again, it comes down to this question: Is the DHCP request required prior to the Linux kernel loading, or is the DHCP request needed after the Linux kernel loads?