L4T 34/35 support for Jetson Nano

Hi,

I would like to understand the main reason of newer L4T not supporting Jetson Nano (regular, not Orin).

This a strategic decision or is there any fundamental technical impediment?

Given the tendency of patches getting upstream, newer kernels from 34/35 should be even more compatible with the previous nano generation, making support even simpler (?).

A) Is this mostly a project decision, focusing HR on newer boards and limiting HR allocated on older platforms?

B) Or is there any key incompatibility, like in the bootloader security chain, etc… that makes newer releases really compatible on older hardware?

I am interested in supporting newer kernel/ubuntu on jetson nano for improved tooling, OTA, etc.

Thanks

I can’t give you a full reason, but I can provide some history. This is long, and all it might do is give you some background. Maybe it will be of interest, don’t know.

The Nano is a TX1 SoC. The TX1 is the very first 64-bit Jetson. NVIDIA has a longer history of Tegra SoC’s, and Jetson is just a subset. The TK1 is the first one with a GPU, and this was 32-bit. This is from over a decade ago. The Nano itself was a newer form factor, but the SoC itself was old when Nano was added. The Nano basically kept alive a dying TX1 market (the other TX1 was enormous in physical size in comparison). This was the first Jetson DIMM format using a lower cost obsolete tech that was still better than what most of the competition had even at their high end.

Then came the TX2. This Tegra SoC was when things were getting significantly better in hardware…it was a second generation 64-bit device with much higher performance than a TX1 (Nano) even though power consumption did not go up.

32-bit had no issues booting with existing Das U-Boot software, but all of the boot up to the U-Boot load itself was and still is quite custom to the specific system (it is the BIOS that makes this uniform and portable, but Jetsons don’t have a BIOS, they only have software; an actual BIOS would consume power, make the unit larger, and make it significantly more expensive). 64-bit boot more or less has to use 32-bit compatibility mode to keep using that software. The 32-bit TK1 used a zImage (compressed) kernel, but 64-bit broke this, and this is when the Image (uncompressed) kernel was used (starting with TX1). The boot chain was obsolete.

If you look at the L4T releases list (L4T is Ubuntu plus NVIDIA drivers…this is what gets flashed) here:
https://developer.nvidia.com/linux-tegra
…then you will find both TX1 and TX2 can use the R32.x (Ubuntu 18.04) release. This was not originally the case, the TX1 had been deprecated at the end of R28.x (R28.x was the first fully stable 64-bit Jetson o/s). However, TX2 was more expensive, and the TX1 Nano came out and so it was revived…R32.x was a second life for TX1 due to the Nano form factor and price popularity. Even so, only the form factor was new. Add a TX2 DIMM form factor and there was a migration path for commercial use. Only the more “hobbyist” market still used the TX1 Nano (or at least so it looked…actual popularity probably put the Nano in more products than expected).

The boot chain itself is something that gets rather complicated since there is no BIOS. It would have been more or less trivial to support TX1 and TX2 with the more recent L4T R35.x if not for that. A lot of improvement went into Xavier boot and UEFI started appearing (both TX1 and TX2 differ so much and are so old this was never migrated). It still isn’t really fully UEFI until you get to L4T R36.x (only Orin can run this; even Xavier is excluded), which is still a developer preview, and it is only now that I suspect future hardware might share enough that when Orin becomes “old” it’ll probably be supported longer than the non-UEFI hardware simply because it provides a uniform boot environment. Hardware dependent boot requirements have finally become “abstracted”.

On top of that boot environment, Orin (L4T R36.x) is the first release using purely the mainline Linux kernel. The out-of-tree code from Xavier and earlier no longer has to be used.

TX1 is just very very old. Supporting migrating this very non-OOP boot is an extreme undertaking. I think that although the Nano has a good place as introductory hardware, it just doesn’t have a future for “new” features. It’s too hard to support. About the only chance for the TX1 Nano to get new features is if someone migrates it to UEFI, and not even Xavier is getting the full UEFI (with stock kernel).

Thank you so much @linuxdev for the detailed response, it’s very helpful.

Apart of newer modules, having a leaner build flow would be a huge bonus (cleaner, more easily controllable, etc).
I’m investigating some approaches base on buildroot/yocto/nix to improve on that sense.

Upgrading hw module and gaining the newer software would be great, but the price bump is quite significant.
Nano: $129 - https://www.arrow.com/en/products/900-13448-0020-000/nvidia
TX2 NX: $199 - https://www.arrow.com/en/products/900-13636-0010-000/nvidia
Orin Nano: $259 - https://www.arrow.com/en/products/900-13767-0040-000/nvidia

Hardware may be older, but from a cost / commercial perspective when targeting a batch release Nano still has relevance, IMHO. It’s unfortunate that we don’t see better sw support (or other hw at the same price range).

Assuming a kernel 6.x has the TX1 driver support (+ patches, DT, etc).
Is it still locked anyhow to the older uboot?
Couldn’t we use the L4T 32.x bootloader but then call a newer Image + rootfs?

I’m still learning about the boot process and this maybe doesn’t even make sense, but could you please suggest some references beyond, nvidia docs, where I could study this further?

Thanks!

The leap in performance of TX2 NX over Nano is fairly large, but the leap from TX2 NX to Xavier Nano or NX was even larger (versus one generation earlier). The Orin Nano is insanely fast in comparison to anything else. If you’re doing something simple then all generations are rather fast. I think even the old Nano has good performance relative to something like RPi, but Orin just blows away all competition. The only competition issues any Jetson has is relative to hard realtime requirements (in which case you might combine a Jetson with a microcontroller if timing requirements are that important).

The kernel of R36.x does not support anything older than the Orin. You could port some of the out-of-tree source code of the original Nano into R36.x…maybe. One important issue of out-of-tree content is related to the X server, and this has more implications than a GUI. The X server itself is just an ABI to a specialized buffer, and because that happens to be an interface to a GPU, people assume this is the GUI. X itself runs only one program, and in most cases this is the desktop manager (and it is the desktop manager which runs those other apps; it is analogous to the kernel running only one program, init, and it is init that runs the other processes).

As a result there is a lot of CUDA type GPU code which actually runs through the X server without having anything to do with being a GUI. If you look at a log from X (e.g., /var/log/Xorg.0.log if your $DISPLAY is “:0”), then you’ll see notes that much of what it uses for talking to it is not coded in X itself, but is instead from other loadable services. For example, all input devices (such as mouse/keyboard) are not coded by X itself, those services dynamically load, and failure to load would break those devices. It is a binary interface and migrating versions requires rebuilding the loadable module binary against the exact ABI of the X server. If you increase the release version of the X server, then you can no longer load the original binary loadable drivers. You would lose a lot of GPU direct hardware access and then only be able to use CPU rendering. CUDA would be lost.

A shorter way of saying that is that if you migrate a system to a newer kernel, then if you want to keep the GPU, you must keep the original X server for the binary loadable module to work. The original drivers are not open source. Some newer GPU code exists which NVIDIA did open source, but I think that migration did not start until roughly the time of Xavier. Holding back the X server release version while migrating everything else two or three generations into the future is difficult. People have migrated older Jetsons to newer software, but so far as I know the GPU is lost except in cases where X is held back.

The R32.x is really now all CBoot, but CBoot has absorbed much of the same interface as UBoot. It only looks like UBoot. Obviously you could use the original R32.x boot chain, but the kernel which inherits this environment must be able to function. Out-of-tree kernel source was used with this, and it is not locked (it is in the source download), but mixing the R32.x boot environment out-of-tree code with a newer kernel may not be as easy as it seems…the newer non-out-of-tree code runs against Orin, which is very different in many ways compared to Nano.

A shorter answer is that you can transfer execution to the Image + rootfs, but significant parts of the kernel might no longer work; the features directly available in the newer kernel are for Orin, not Nano, so you’d be making that migration yourself.

The official documentation for a specific L4T release is best, but may not have all you want:
https://developer.nvidia.com/linux-tegra

The general documentation has more documentation, some of which is for people developing new carrier boards or customizing the kernel, which can be found here (and you can limit docs by which Jetson model you have):
https://developer.nvidia.com/embedded/downloads

You’ll find a lot about bringing up new carrier boards with new features, and for use with various custom peripherals. What you won’t find is detailed information on the module itself…the module schematic is a closed black box. Information on what a module must interface with though is there.

Keep in mind when you read that documentation that a device tree is more or less a set of arguments to pass to a kernel driver as it loads. Often this says where to find the hardware (e.g., a physical address), and the rest of this tree content is specific to its exact driver. If you migrate to a newer driver, then it is possible the driver has changed and that tree fragment may no longer apply. A really good way to understand how difficult this can be is to take a device tree fragment for a TX1 Nano and compare it to the same device tree fragment of the same driver in the newer mainline 6.x Linux kernel…that fragment difference is a clue to how the driver has changed. If the driver is new (a fragment might be used by more than one driver…more on that below), then the tree fragment itself might be completely missing in the older Nano.

Device trees are considered firmware. Boot chains are actually a series of micro operating systems setting up the hardware in a certain state before running its one program…the next boot stage…and ending that stage’s life. There is an MB1 boot stage, MB2, then CBoot, and finally the Linux kernel. If you go back far enough, then CBoot loaded UBoot instead of the Linux kernel. Each stage might need a different set of drivers depending on what is being accessed during boot, and it is possible each stage needs firmware/device tree. Now imagine you’ve migrated everything in the Linux kernel, and you have a shiny new device tree. What about the drivers in the earlier boot stages, and any device tree content for those stages? Were those also migrated? Or were those drivers using the old tree, and the old drivers are performing a valid setup relative to what the Linux kernel wants?

This isn’t particularly useful as more than an illustration, but have you ever noticed that the serial UARTs available to developers tend to have device special file names such as “/dev/ttyS0” and “/dev/ttyTHS0”? The Tegra UARTs have multiple modes available since they were invented so long ago (far longer in the past than the GPU was added), and they were given the ability to use either more modern variants of emulation (the 16550A), or older UART (16450) emulation. Each device special file is not a real file, those files exist in RAM and are part of the driver. The driver to /dev/ttyS0 and /dev/ttyTHS0 goes to the same physical UART, but both drivers are available to that single UART. Why would that be done? Well, I’ll continue that story since it shows how boot and Linux stages can have multiple drivers…

The boot chain was always UBoot in the early days. UBoot has drivers for the 16550A. The ttyTHS0 driver though has DMA available which drops the requirement for CPU participation when transferring larger amounts of data. In order to use DMA in boot stages one would have to port the ttyTHS (“THS” means “Tegra High Speed”) to UBoot in earlier boot chains. Then one would have to migrate it to CBoot. UBoot would need custom maintenance after that, and you’d no longer be using mainline UBoot. Now imagine you are using that UART for boot logging, including boot stages before Linux ever loads.

What happens when we transition from a ttyS driver to a ttyTHS driver? The hardware might reset, and log messages might be lost. You will find that most of the time serial console will simply use the ttyS driver…this guarantees continuity of serial console service when transitioning between environments. The UART stays in a valid state since different drivers are not swapping out on the same hardware. Boot software no longer requires custom maintenance with the ttyS driver.

If you want to experiment, then you could experiment with porting the publicly available ttyTHS driver to the boot stages (at least you could port to CBoot…porting to MB1 might be problematic, it isn’t so well documented). Finding a list of every driver used in the boot stages would tell you which drivers are potentially a problem when transitioning to the Linux kernel in a kernel which is an entire generation different. Not only do you need to migrate a 4.x kernel into a 5.x kernel, you now also must migrate it to the boot chain. Maybe CBoot migration is enough, and probably it is for most hardware (MB1 and MB2 might not have need of the new driver).

Also, remember how I said the newer models use UEFI? UEFI has a hardware-dependent earlier stage. Once that is done, UEFI has an abstracted uniform interface of services it provides. Any boot software which uses standard UEFI services can easily be dropped in place (it might still need device tree edits). But all of the older Jetson releases are not UEFI. So now you would have to take anything UEFI and port it to CBoot/UBoot, or else write your own custom UEFI layer in the boot chain. Then you could have a standard UEFI boot on an older Nano using the newer kernel. It took a couple of years to completely replace the older boot chain on newer hardware with purely UEFI. That would be up to you for parts of software in the boot chain that the newer kernel depends on. A lot would work without that, but the few things which care would not be trivial to port.

So it isn’t just about a bootloader. It is about what leads to setup of the boot environment before the bootloader overwrites itself with the Linux kernel. Obviously it is possible, NVIDIA has done this, but that migration started in Xavier and is fully supported in Orin (and is still preview tech in Orin, though that will change soon). It isn’t too hard to load the kernel and rootfs. It is difficult to get all of it working without more customization than you might think. You’re developing BIOS software too, not just bootloader. In some cases you will just get lucky.

A place to start is the Bring-Up guide. See:
https://developer.nvidia.com/embedded/downloads#?search=bring-up&tx=$product,jetson_nano
…but this is more about customizing carrier boards (which in turn is important during boot stages).

The general Nano docs:
https://developer.nvidia.com/embedded/downloads#?tx=$product,jetson_nano

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.