Unable to update U-boot environment variable "ramdisk_addr_r" after flashing with JetPack 3.2.1

Hello,

I recently updated my TX2 and TX2i modules with JetPack 3.2.1 (since JetPack 3.2 is no longer avaiable). I built my custom kernel and built an initrd.img file. But when I boot the system with the new kernel Image and initrd.img files, it gets stuck with below message:

Trying to unpack rootfs image as initramfs…
rootfs image is not initramfs (junk in compressed archive); looks like an initrd

Usually, this is resolved if I set U-boot environment variable “ramdisk_addr_r” value to 0x92800000 instead of current 0x82800000. This way the large size of Image file doesn’t interfere with the initrd.img file, which I think is the cause of it. $kernel_addr_r is overwriting $ramdisk_addr_r or vice-versa.

In order to boot the kernel, I do “setenv ram_disk_r 92800000” on the U-boot prompt and run “saveenv”. It does show me that the value has been written to the eMMC (0) part. However, if I reset the board, the value is back to 0x82800000. The value is in alignment with “ramdisk_addr_r_align” of 0x0020000.

I am not sure if this has happened on JetPack 3.2 as well because I can’t go back to that. But it had worked on JetPack which installed l4t version r28.0. Please let me know if I am doing something wrong here or is there a different way to set a custom value to “ramdisk_addr_r” env variable.

The default L4T does not use an initrd. I don’t know if there are any issues in using the initrd, but keep in mind there isn’t much experience with using this on a Jetson. The FDT device tree entry in extlinux.conf definitely won’t work as expected. If your initrd has device tree specified, then this too would be an issue even if the initrd otherwise works as expected.

Normally you set the variable first, and then saveenv after (resulting in saving the value over reboots). On the other hand, if the device tree also sets this, then it may be upon boot your edit was there until the device tree loaded (prior boot stages edit the device tree and then pass it along to the next stage…if the variable is initialized via the device tree then the tree may take precedent as the most recent edit). I have not looked to see if this is actually the case, but I would suggest first investigating if the device tree also specifies this value.

Thank you, linuxdev for the response. I will take a look in this direction.

Note that you can get a reverse compile of an existing device tree this way:

dtc -I fs -O dts -o extracted.dts /proc/device-tree

(or browse “/proc/device-tree”, but an editor is easier to find things in)

I did this and searched for “ramdisk_addr_r” as well as “initrd”. Saw only below snippet:

chosen {
nvidia,bluetooth-mac = “00:ff:ff:ff:ff:ff”;
stdout-path = “/serial@3100000”;
board-has-eeprom;
linux,initrd-end = <0x0 0x8314c5cf>;
nvidia,ether-mac = “00:04:4b:ab:f8:65”;
nvidia,wifi-mac = “00:ff:ff:ff:ff:ff”;
bootargs = “root=/dev/mmcblk0p1 rw rootwait console=ttyS0,115200n8 console=tty0 OS=l4t fbcon=map:0 net.ifnames=0 memtype=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x03100000 nvdumper_reserved=0x2372e0000 gpt tegra_fbmem2=0x140000@0x969ee000 lut_mem2=0x2008@0x969eb000 tegraid=18.1.2.0.0 tegra_keep_boot_clocks maxcpus=6 boot.slot_suffix= boot.ratchetvalues=0.2.1 androidboot.serialno=0334817020277 bl_prof_dataptr=0x10000@0x237040000 sdhci_tegra.en_boot_part_access=1 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4”;
linux,initrd-start = <0x0 0x82800000>;

I have a TX1 which was flashed by JetPack 3.2. There also I extracted the /proc/device-tree and found similar snippet for initrd. But there I can change “ramdisk_addr_r” environment variable through U-boot prompt. But that may not be exact comparison since they both use different dtbs and JetPack 3.2.1 updated only TX2 and TX2i stuff. TX1 is still r28.0 tree.

Some things I can try:

  1. Use a TX1 flashed with JetPack 3.2.1 and see if I can change the variable.
  2. Look at the u-boot source code for TX1 and TX2, if 1. works otherwise compare u-boot source code between JetPack 3.2 and 3.2.1 for any differences.

I am thinking about U-boot here because the value of the variable is changed only over the reboot(reset) and isn’t if I continue without reset after changing the variable. (e.g. “run bootcmd_mmc0”).

I know U-boot uses its own Image and dtb file which is used to initialize various devices on the module required for U-boot to correctly boot, e.g. network interface, filesystem, etc. Please let me know if this approach sounds wrong.

I’m going to reverse course a bit and ask what is it that you need the initrd file for? What makes this so big?

Next, an observation with some history: Normally modules and the initrd share a space in physical address directly below the start of the kernel. The maximum size of a direct branch instruction is what limits the total size. On a 32-bit ARMv7 this was limited to 32MB. I think on 64-vit ARMv8 it grows to 128MB. Within this range of physical address below the start of kernel anything occurring outside of this results in “unreachable code” with undefined side effects. Within this region there is some preset location to define where initrd and modules separate. This isn’t dynamic, it is a fixed location. On 32-bit ARMv7 you had these two mutually exclusive options which defines the split:

CONFIG_TASK_SIZE_3G_LESS_16M
CONFIG_TASK_SIZE_3G_LESS_24M
(EDIT: I'm not actually positive if this is the correct config but probably it is)

On arm32 picking 16MB for one implies 16MB for the other. Picking 24MB for one implies 8MB for the other. If initrd or module combined sizes exceed its limit, then it is essentially a buffer overflow of one into the other.

Picking a combined total size and offset of initrd in the wrong direction would imply the kernel itself may be overwriting the initrd space (or more accurately, the bootloader might be loading the kernel into memory which clobbers the upper address area of your initrd). More available space would imply a lower base physical address since this space preceeds the kernel’s starting address.

I am not sure where that dividing line is for the 128MB direct branch limit of 64-bit ARM, but the placement at 0x92800000 instead of current 0x82800000 might be the opposite of what you think it is since this moves memory closer to the start of the kernel instead of further away. I also don’t know simply putting this at a different address would be interpreted correctly if modules do not also know about this address change (I have neither tried nor researched this). A lot could be simplified if your initrd wasn’t that big. Do you really need this for boot to start (e.g., is it for filesystems)?

Going on to the original question, I did this experiment. I created an environment variable “deleteme_var” set to value “abcxyz” in the U-Boot console. I then ran “saveenv” and restarted without running Linux itself…the variable still existed. Then I booted completely into Linux, and again rebooted back into the U-Boot console. Upon reboot after going through the full Linux boot cycle the variable was still there and correctly set. This tells you how memory is saved when there is nothing overwriting it.

The environment variable ramdisk_addr_r itself is likely not being changed through any U-Boot mechanism forgetting or losing the edited value. Very likely there is some other part of boot editing this before U-Boot gets its environment passed to it. Perhaps something in cboot…I don’t know. Perhaps the device tree…I don’t know. Perhaps it is just a sanity test being performed which doesn’t like this value and is reverting it to a default…I don’t know.

If you really need this address change for a bigger initrd I’ll suggest that you need to research what constraints there are on arm64 initrd/module address space setup. Even if your environment is able to save an edited ramdisk_addr_r it is likely the kernel itself would also need to know about the change (and it is architecture specific how to do this). If that total space from the base address was already 128MB, then there is no possibility of extending it further due to limitations of the direct branch assembler instruction. If the space reserved was less than or equal to 128MB, then perhaps what you need to do is edit the dividing line of how modules and initrd share that space since the total can never exceed the direct branch size (and I don’t know what mechanisms exist under arm64 to do this).

I am building a debug and prt kernel which turns on some high CONFIG_ options for debugging and PREEMPT_RT kernel. This is one of the kernel flavors that www.concurrent-rt.com provides.

You may be right about the ramdisk_addr_r value to be set to lower value instead of higher but I see that the U-boot environment variables kernel_addr_r and fdt_addr_r both are of lower value, 0x8008000 and 0x82000000 respectively. So, if I lower the value of ramdisk_addr_r, I may end up reading the values at the addresses where either kernel or fdt might be residing. And that was the reason for my approach of increasing the value. I may be wrong and more study will be required.

AFAIK, there have been changes to the cboot code from r28.2 to r28.2.1. And that might have caused this bug (if it is one). I grepped for certain terms in the cboot code but nothing useful is found so far. I am hoping that one of the NVIDIA developers comment on it.

I made sure that TX1 flashed with JetPack 3.2.1 doesn’t have this issue.

Edit: Ohh and one more data point is that I was able to change values of other env variables or add new variables on TX2/TX2i flashed with JetPack 3.2.1.

This doesn’t answer your question, but here’s some observations from an experiment where I named an empty initrd file via the extlinux.conf INITRD key/value pair and logged boot (R28.2, TX2, so I am missing any differences between my R28.2 and your R28.2.1…I’ll probably flash R28.2.1 this weekend).

Prior to reaching U-Boot console I observed this:

[0003.350] I> Copying kernel image (488466 bytes) from 0xa8000800 to 0x80080000 ... [0003.357] I> Done
[0003.359] I> Move ramdisk (len: 0) from 0xa8078000 to 0x9d000000
[0003.366] I> Updated bpmp info to DTB
[0003.372] I> Ramdisk: Base: 0x9d000000; Size: 0x0
[0003.376] I> Updated initrd info to DTB

Notice that ramdisk operations began prior to U-Boot ever being reached. For reference, the base address was 0x9d000000, and size was listed as 0 bytes. However, the ramdisk being considered by the loader at this point in time does not correspond to the one at “/boot/initrd” (this same initrd line is logged even when extlinux.conf does not have an INITRD entry). Since this is an early pre-U-Boot stage I suspect this was the result of trying to find an initrd in a less tradition location: A partition. It would be useful to find out why and how pre-U-Boot stages became interested in an initial ramdisk. Anyone know details of the pre-U-Boot initial ramdisk load? Is there some new scheme to load this from a partition instead of from a file in “/boot”, or as the original poster asked, are any details available on how this may have changed going from R28.2 to R28.2.1?

The address 0x9d000000 of the pre-U-Boot initrd load log is never again mentioned after that (perhaps because of a 0 size). My fake initrd (one of 0 size) also does not say what address it was loaded at, so the experiment wasn’t very useful.

During your initrd testing did you ever have a smaller initrd which worked? If so, can you post the serial console log from power on until when Linux actually begins running? I am curious what addresses were being logged.

At the base/parent location of your current failing initrd directory tree what is the exact total size (you can use “du -s .” from the root of the directory)? I am interested in seeing any load address information in a standard working case.

For the file system within the initrd you are trying to use can you post the output of the “tree” command (you might need to “sudo apt-get install tree”)? I am interested in this as a way of reproducing the issue (I might create an initrd based on that…both one smaller and larger than the default space allows).

Hi linuxde,

I am able to boot an initrd of smaller size. In fact, I am able to boot with an initrd.img whose corresponding kernel image is only 3000bytes smaller than the one which is not working.

nvidia@drax:/boot$ ls -ltr initrd*
-rw-r–r-- 1 root root 0 Jul 12 18:47 initrd
-rw-r–r-- 1 root root 9491730 Jul 13 08:59 initrd.img-4.4.38-rt49-r28.2.1-RedHawk-7.3.2
-rw-r–r-- 1 root root 9494851 Jul 13 08:59 initrd.img-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-prt
-rw-r–r-- 1 root root 9742543 Jul 13 09:00 initrd.img-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-prt-debug
-rw-r–r-- 1 root root 9495638 Jul 13 09:00 initrd.img-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-prt-trace
-rw-r–r-- 1 root root 9741052 Jul 13 13:06 initrd.img-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-debug
-rw-r–r-- 1 root root 9749967 Jul 13 15:26 initrd.img-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-trace
nvidia@drax:/boot$
nvidia@drax:/boot$ ls -ltr Image*
-rwxr-xr-x 1 root root 18604128 Jul 12 08:11 Image-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-trace
-rwxr-xr-x 1 root root 25195232 Jul 12 08:33 Image-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-debug
-rwxr-xr-x 1 root root 18529504 Jul 12 08:54 Image-4.4.38-rt49-r28.2.1-RedHawk-7.3.2
-rwxr-xr-x 1 root root 15482592 Jul 12 09:13 Image-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-prt-trace
-rwxr-xr-x 1 root root 25400032 Jul 12 09:35 Image-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-prt-debug
-rwxr-xr-x 1 root root 15407840 Jul 12 09:55 Image-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-prt
-rw-r–r-- 1 root root 20727600 Jul 12 18:47 Image
nvidia@drax:/boot$

So, I am able to boot with all the initrd* files except “initrd.img-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-prt-debug” file, and corresponding Image*-prt-debug kernel image. If you observe initrd.imgprt-debug size is smaller but Imageprt-debug is biggest. I hope it’s not confusing.

I have attached boot log when booting with “Image-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-debug” and “initrd.img-4.4.38-rt49-r28.2.1-RedHawk-7.3.2-debug”. I have also attached the output of “tree” command.
NVIDIA-devtalk-debug-initrd-log.txt (179 KB)
tree-output.txt (35.7 MB)

If you look at an initrd it is just a cpio archive. The file sizes I’m interested in would be the total content of the unpacked initrd (this is what the size limits are on, not the initial package).

Unfortunately, once U-Boot was reached I didn’t see any added notes to verify what address the initrd was loaded at (I only saw virtual address notes and not physical address and it is the physical address I wanted to verify).

To unpack a given initrd into its directory tre you can do something like this inside of an empty directory:

  1. Uncompress:
    gunzip < /where/ever/it/is/initrd > initrd.cpio
    
  2. Expand:
    sudo cpio -vid < initrd.cpio
    

Normally the only content mandatory for an initrd disk would be modules which must be loaded in order to read the file system. Traditionally this would mean modules for ext4 if not built in natively to the kernel. If you have an ext4 module, then you could remove it if the kernel gets ext4 built in natively. Another typical example would be a RAID or other disk controller in module format…and once again, those could be removed from the initrd if they are either built directly in to the kernel, or if the initial file system is not RAID (I run RAID1 on “/home” and “/usr/local” and “/var/www” on my PC…I don’t need an initrd for this because my “/” is not RAID).

If your initrd is that close to being able to work, then I would expect it to be quite possible removing just one unnecessary item would remove the whole customization issue.

A typical way to create an initrd follows. I’ll assume you have the original initrd with some edit and you are in the root of that directory attempting to create an initrd in the parent directory to that:

  1. ``` find . -print | cpio -ov > ../initrd-modified.cpio ```
  2. ``` cd .. ```
  3. ``` gzip initrd-modified.cpio ```
  4. ``` mv initrd-modified.cpio initrd-modified ```
  5. "rm" any of your unneeded original working tree.
  6. Alternate single line command from the top level of the tree creating the initrd:
    find . | cpio --quiet -H newc -o | gzip -9 -n > ../initrd_new
    

What is the “tree” output of one of the INITRDs which is too large?

Hi linuxdev,

I was able to figure out the reason behind this. With my code (majorly for x86_64 ported to ARM64), I had increased the size of the lockdep structures. But since the memory on these boxes is not much this huge lockdep structures caused, probably overrunning the memory during kernel load.

When I say structures I mean the number of lock chains to look at and couple other things.