Booting 28.1 from 32.5.1 U-boot

To support software downgrades for my company’s application, I would like to be able to boot a 28.1 kernel (running Linux 4.4) from the newer version of u-boot (based off of 6b630d64fd86, circa 2020). Has anyone else attempted anything similar and if so do you have any recommendations?

This would be done to mitigate the risk of bricking field devices if anything goes wrong during a downgrade. Unfortunately, with this new version, it seems like after u-bot logs “Starting kernel …” nothing else is logged to the serial terminal. The /chosen/stdout-path device tree variable is set appropriately, and the bootargs specify console and earlycon arguments (indeed, all of the same kernel cmdline arguments) that work when booting the 4.4 kernel from the 28.1 u-boot version, but for some reason it doesn’t seem to work when booting from the newer bootloader.

I have changed the configuration to perform the early board init, so the cboot arguments and fdt path are passed to u-boot. The kernel appears to be loaded in the correct memory location and examining the first few bytes shows that the image header is unchanged. the fdt command in u-boot shows that the device tree is also intact and uncorrupted. There are no error messages logged that show failures, even with the logging level increased.

Any help is appreciated, it’s been a difficult one ti try and figure out.

There wouldn’t be any support for this since mixing releases (as you have found out) usually results in a failed system. There are strong dependencies. I will suggest the mixing of R28.x with R32.x kernels is unlikely to have a satisfactory result, and this is probably a “bad idea” (especially since boot content packages in R32.5.1 might update and you’d probably need to disable this update and test each update if you manage to get it to work). However, and I can’t make suggestions beyond what is below, here are some things to think about (this is more or less just because it is an interesting question, not because I think this would succeed, but answering these questions would greatly improve understanding how boot works/fails when mixing releases)…

You might consider that the failure is perhaps in an initrd (if used) rather than the main rootfs. Don’t know, just suggesting initrd is something to consider. Examine initrd of the R28.1 system and compare to the R32.5.1 system. For example, make sure it is able to find the modules for the correct kernel, both for what it loads in the initrd, and for the rootfs at “/lib/modules/$(uname -r)/kernel”.

Also note that the kernel itself is PID 0, and the first thing the kernel runs (once it finishes loading) is init (PID 1). I suspect that the content of init also changes between R28.1 and R32.5.1 (probably by a lot), and if the kernel works, but init is wrong, then it would simply stop where it is at.

Parts of init have to be considered within any initrd as a way of passing a working environment to the actual rootfs (for example, if the ext4 filesystem driver is in the form of a module, then the module must exist in the initrd before the full rootfs can ever load). Then the init which is part of the rootfs (and not initrd “miniature” init) would also have to work with that version of init (R28.1 and R32.5.1 have very different init).

If you are not using an initrd in the newer release with older kernel, then you might be able to write an initrd “adapter” should you find some incompatibility between the two versions of init.

Very likely you will want to disable any “quiet” argument for logging, add printk statements to say when various init stages have been reached (especially the very first line of init), and figure out if it is really init which is failing. I recommend practicing on a working R32.5.1 system and seeing if you can get serial console to echo log lines to the console at various stages of the init software (distinguishing between initrd init lines and rootfs init). You probably must entirely use all of the R32.5.1 boot content up until kernel load and init (except perhaps some edits to device tree or environment variables).

Another thing to check is what you see in “/proc/cmdline” for a purely R28.1 system versus the R32.5.1 system. If you look at “/boot/extlinux/extlinux.conf”, then you’ll see that kernel arguments start by inheriting from the bootloader via the “${cbootargs}”. Read up on boot chains and see if you can find the entire origin of what ends up in cbootargs and any differences to what you see.

Remember that an argument to the kernel is visible to any kernel module or driver, and that the arguments which work for an R32.5.1 kernel might differ from what works for an R28.1 kernel.

Note that the sources of what is inherited in cbootargs (how loaded and from where) may differ in older boot content versus newer boot content. You’ll probably need to go through docs for both releases to understand how this is cbootargs variable derived. Also, realize that earlier boot stages can edit what the source of cbootargs is prior to passing it to the kernel (so for example, if you know values in this variable start in a partition, you don’t know if that content will still be an exact match when it reaches cbootargs for the kernel to inherit). I don’t know if that content would have an effect stopping an older kernel from working, but if you were to use R28.1 boot content to reach the kernel cmdline, then boot probably would fail or behave oddly.

Unfortunate, but understandable.