The install was easy enough. You simply untar the package in your L4T directory and run the simple flash.sh command noted in the documentation. Don’t try it until you read all the notes below!
Notes:
I was NOT able to get the NX to boot using the ACPI version of the firmware. The device tree version worked fine. See the device tree note below though.
As noted in the documentation, the nvidia L4T 4.9 kernel will NOT work with UEFI so you don’t get any of the nvidia specific drivers. I tried to add the UEFI support and recompile the kernel but nothing seemed to work. Grub just failed to load the kernel image. This means that none of the Jetson specific stuff like nvpmodel, jetson_clocks, etc. will work.
The current mainline kernels work fine. I’m using the latest Fedora built kernel which is 5.11.14.
Again, as stated in the documentation, you MUST add the earlycon kernel command line parameter if you want serial support. You’ll need the serial console connected anyway to interact with the UEFI settings and grub.
The documentation has a section called “Booting L4T with mainline Linux (Jetson AGX Xavier only)” but NX users will also need to follow some of the steps, especially if you want stable network connectivity. The Fedora built kernel includes the correct ethernet drivers but has the contiguous memory allocation size set at 64mb. This is NOT enough for a stable connection under load. Luckily, you can set cma=256M on the kernel command line to increase it to the proper size.
Speaking of command line… Fedora has the “fedora-arm-image-installer” whichs allow you to create a boot drive for arm architectures. You can specify the additional kernel parameters when you create the image. Not sure about other distros but they probably have the same thing. It worked fine to create a Fedora disk image.
Interacting with the UEFI configuration was straightforward. Be aware though…if you reflash, the UEFI settings go back to default. It’s not really a big deal since the defaults are sane. The only thing I would up altering was the reduction of the boot wait timeout from 10 seconds to 2. With just an NVMe stick installed on the devkit carrier, the firmware detected the EFI partition automatically but if you have more than one storage device attached, you’ll need to look at the entries and set the boot order.
Speaking of NVMe… The stock linux kernel does NOT detect some NVMe brands on the pcie controller. I spent almost 2 days trying to figure out why my Crucial sticks weren’t being recognized. When I tried a Samsung stick, it worked fine. The Crucials did work fine with the nvidia 4.9 kernel. Not sure if this has something to do with the PCIe support mentioned at the end of the documentation. If I have a chance this week, I’ll try a kernel compile with the PCIe patches and see if the Crucial sticks get recognized.
Device tree support is minimal and you can’t use any of the existing dtbs or overlays. Although Grub2 itself is supposed to have support for specifying dtbs and overlays, the Fedora version doesn’t so the only way to get additional stuff into the device tree is to dump the uefi-specific one at L4T/kernel/dtb/tegra194-p3509-0000+p3668-0000-uefi.dtb back to dts, modify it, then recompile it back to a dtb and flash again.
Having said that, there appears to be a bug in flash.sh that causes the -d option (provide dtb filename) to be ignored. You either have to name yours the same as the default one (/tegra194-p3509-0000+p3668-0000-uefi.dtb) or modify the DTB_FILE entry in p3509-0000+p3668-0000-uefi-qspi.conf.
If you are going to create your own, you can’t just copy and paste from existing dts files into your new dts file. The structure of the uefi file isn’t the same as the existing structure, there is no symbol table, and the phandles and device names have changed. It took me another half day just to get my pps-gpio device added. I’ll be adding spidev support shortly.
That’s it for now but I’ll update this post with more info as I discover stuff.
Oh, someone from nvidia might want to add a “uefi” tag to the forum.
I am curious why this is, becase we have validated this and tested on both Jetson AGX Xavier and Jetson Xavier NX. Please note that when using ACPI the Linux console output is on a different serial port than the default used with L4T. You may have already seen this, but there is some information in the section Linux Serial Console of the README.
This is correct and unfortunately expected at this time.
Adding earlycon should not be necessary, but adding the ‘console’ information to the kernel command line when booting with device-tree is.
Thanks for the feedback. Can you let us know the exact Crucial NVMe cards you are using? Please note that the PCIe patches listed are ONLY required for ACPI. These should not impact device-tree.
With Fedora, you should be able to edit the menu item for booting Fedora (by pressing ‘e’) and specify the DTB you are using in the use the ‘devicetree’ keyword …
load_video
set gfxpayload=keep
insmod gzio
linux ($root)/vmlinuz-5.8.6-301.fc33.aarch64 console=ttyS0,115200n8 root=UUID=b9c84f8b-74cc-4615-b8bb-59eee5ec46b7 ro
initrd ($root)/initramfs-5.8.6-301.fc33.aarch64.img
devicetree ($root)/dtb/nvidia/my-jetson.dtb
This does work fine for Jetson AGX Xavier. However, as you previously mentioned and per the README, because the instructions for booting maineline Linux are only supported for Jetson AGX Xavier, we are missing some additional flash configurations for NX to allow you to flash/boot mainline Linux with the L4T rootfs. I understand that you were able to workaround this and so top points for getting that to work! However, we do plan to address this and make this easier. Hopefully, then the -d option should work as expected.
Thanks again for all the feedback. Very much appreciated!
Actually, I didn’t see the note about ttyS0. :) I’m so used to using the port on the front header. However, when using ACPI, the kernel hangs looking for the root device with or without enabling PCIe in the firmware.
ACPI:
pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
pci 0005:00:00.0: PTM enabled (root), 16ns granularity
pci 0005:00:00.0: PCI bridge to [bus 01-ff]
pcieport 0005:00:00.0: PME: Signaling with IRQ 51
pcieport 0005:00:00.0: AER: enabled with IRQ 51
pci_bus 0005:01: busn_res: [bus 01-ff] is released
pci_bus 0005:00: busn_res: [bus 00-ff] is released
tegra-xusb 3610000.usb: Firmware timestamp: 2019-07-24 05:47:34 UTC
tegra-xusb 3610000.usb: xHCI Host Controller
Yep, you’re right. Using ttyS0 and the pins on the 40-pin header does work. Kinda confusing though because many folks using the devkit are used to using the front header for console and the 40 pin header for other serial things. Not a big deal though.
The hang with the Crucial sticks using devicetree is the same as the ACPI hang mentioned above. The PCIe controller gets initialized but not the nvme device. BTW, it was the Samsung I was testing with ACPI.
Here’s the Samsung for reference:
Oy. There’s a comment in Fedora’s documentation that says “devicetree” isn’t supported but I tried it anyway and it didn’t work. What I should have done is check the file name i specified because I had a typo in it. I just fixed the typo and you’re right, it does work. Sigh.
Cool. Now that I fixed my typo and found that devicetree does work, this isn’t an issue. :)
Well, once I fixed my typos and read a little more carefully, things are looking even better!
Have you tried ACPI with the Fedora rootfs? If you write the Fedora image to the NVMe card and boot with ACPI it should work.
Yes it is annoying I agree. The problem stems from the fact that ACPI does not have support for the TCU (Tegra Combined UART) driver. For booting with ACPI we use the Tegra 8250 driver. Now although it is completely possible to use the Tegra 8250 driver on the physical UART on the front header, it does involve some additional flash time configuration changes which are not ideal either. We wanted to be able to flip between device-tree and ACPI at boot time without needing to reflash and hence, ended up using a different UART.
I tried Fedora Server 34 today on Jetson Xavier NX with NVMe and I was able to boot it fine using ACPI. Please ensure that you have enabled PCIe support via the ‘Device Manager’ menu as detailed in the Enabling PCIe Support section of the README. Please note that the Fedora kernel already has the PCIe kernel patches that are referenced in the README and so there is no need to rebuild the Fedora kernel. The stock kernel should work fine. I am also attaching a boot log from my Xavier NX.
Thanks @jonathanh. I’ll give it a shot again today. I probably missed something.
One more question… Can you guys share the source dts files for the uefi dtbs? I did dump the uefi dtb but it helps to see the source with all the symbols and defines in tact.
Another goof on my part :) I didn’t remove the devicetree entry in Grub after setting the firmware to ACPI. I got booted OK except…
I never got an hdmi console. I didn’t have time to troubleshoot that but I can.
The wonderful (In-)Consistent Device Naming troublemaker renamed my network interface from “eth0” to “enanvda1160i0” which systemd-networkd did not like since I was matching on device name. A quick switch to match on mac address fixed that though.
Hardware monitoring was mostly missing. There was only an hwmon node for the nvme and there was one cooling device for the cpu fan. The only control there is to set the cur_state. I’m not sure if I read something about this somewhere or not.
Oh yeah… Is there any downside to using the ACPI firmware all the time and just switching the device mode back and forth between ACPI and Devicetree?
Back to serial support. I was wrong about it being workable because you can’t permanently connect anything to the TCU port. I originally had my GPS unit connected to the UART1 pins on the 40 pin header. WIth that now being the console, I tried using the TCU port for the GPS and all hell broke loose. Since TBoot still dumps its output to TCU, it confuses the GPS unit AND since the UEFI is still both outputting and taking input from the TCU, the UEFI menus and Grub go wild.
One solution would be for me to keep the GPS’s reset pin low via GPIO then set the pin to high after the boot is complete. Another might be to use I2C or SPI as the communication channel but then I’d need to create a bridge to expose them as serial ports. Neither is particularly appealing nor are they plug-and-play for other users.
The last solution, which is what I’ll probably wind up doing, is to use the device tree to reassign two of the other pins on the 40 pin header to another UART. This is where having the original dts, dtsi, and header files for the UEFI dtb would really help.
That’s easy enough for me but not really portable either.
Sorry for the delay, I was checking on this. The DTB is essentially a modified version of the device-tree sources for Jetson Xavier NX found in the mainline kernel. The following changes have been made to this …
The changes were needed for one of the initial bootloaders that runs before UEFI starts. We are working to eliminate these changes and just use the upstream version.
If you are using device-tree and not ACPI, then probably the simplest solution is to use the TCU for console, and not the UART on the 40-pin header. There should be nothing to prevent you from doing that. It is more of a problem when using ACPI.
Thanks for the device tree info. There does seem to be a lot of things missing from the uefi version. Is that stuff for which there’s no driver yet? Also a lot of stuff moved below the bus@0 node. Was that also required for kernel 5.x compatibility?
If you are comparing with the DTB that is shipped for L4T, yes there are differences, however, this is simply because we are using the device-tree from the mainline kernel source tree. The focus here has been on booting mainline Linux kernels and hence it made sense to align with the mainline device-tree.
I’ll probably be doing some devicetree work in the near future (spi at least). Since you’re the maintainer, what branch should I develop against? I was thinking for-5.13/arm64/dt assuming I get them to you in time. If that doesn’t make sense just let me know.
Also, was the tegra_gte driver one of the contested ones? I want to try and get that working as well.
Are you planning to submit these to the upstream Linux kernel branch? If so, then we typically just develop on top of -next. The changes for v5.13 have been submitted now and so once the merge window closes for v5.13 (in about a week), changes for v5.14 can be submitted.
What do you mean by contested? Ideally, it would go upstream but there are some discussions going on about how best to go this.
-next it is. I haven’t submitted kernel patches in a very long time so I’d prefer to send them to you, properly formatted of course. Things could get very confusing if I sent them up myself while you guys are also doing it.
Sorry, this was from another thread…
The module I want to get updated first is tegra_gte because I use for precision timekeeping.
EDIT:
HA! I just looked at the “discussion” link in your post which was for gte. :)
EDIT2:
Just for reference, I use the GTE/AON GPIO with pps_gpio.
@gtj, @jonathanh,
I’ve created UEFI with Jetson NX Xavier - v1.1.0-26613629 in order to avoid poluting this thread. If any of you happen to have time, please, take a look at that, as my experience was not exactly the same as the one described here. :-).
