Flash only RootFS into NVME and kernel,bootloader into EMMC. JP 4.6

Thanks.
I don’t have direct access to the nvme (its a custom board) so I have to use the initrd “method”.

The problem is that I didn’t find explicit instructions regarding splitting the rootfs from the kernel and bootloader.

Maybe there is a gap in my understanding, but from what I understood the system.img contains the rootfs, kernel and bootloader. So if I flash it to a specific location I flash all the three to the same place.

So what I’m looking for is a command which selects a destination for the kernel and bootloader and a different destination for the rootfs. In addition it has to edit some configurations in order for the kernel / bootloader to recognise the rootfs in its different location.

Thanks

No, system.img only has info for APP partition… which does not have bootloader.

And currently we only support to load bootloader from internal drive. Which means QSPI-NOR flash on your NX.

If I understood correctly:

  1. All the flashing tools (flash.sh / initrd and more) will flash the bootloader into my eMMC.
  2. system.img contains kernel and rootfs only
  3. the " Flashing to an NVMe Drive" chapter, discuss on flashing the rootfs and kernel only to the NVME

Is that correct?

Actually, it is QSPI-NOR flash on your jetson module which has the bootloader. Not emmc (for NX only)

2 and 3 are correctly…

I would suggest you can firstly dump the uart log first to understand the boot flow… otherwise, it would be very difficult to debug…

Wayne thank you for your patience and answers.

I do look at the uart log during bootup (but , for now for me , its kinda messy and complicated) like the flash log…

If I want to split the kernel and the rootfs, flash them into different devices. Is it possible?

it is possible and you don’t even need to run initrd flash(depends on your usecase)…Please read my post here first…

If you have something which you cannot understand, please ask. Thanks.

I read everything and have a couple of questions please:

  1. You wrote:

Blockquote
3. How to do?
We provide steps inside the developer guide link for booting both kernel and mount rootfs from the same drive.
Please be careful that every platform shares different bootloader capability. For example, TX2 uboot does not support usb .
If you just want to mount the file system from external, then you can check this page.
I am sorry that the title of this page is not precise. It is just mounting the rootfs from external. Not booting from anything.
https://elinux.org/Jetson/L4T/Boot_From_External_Device

1.1. This page explains how to boot from external device but only in case its connected to your host first. I can’t do that, therefore I have to use the NVME flash with “initrd method”.
The problem is that with this method I flash both rootfs and kernel into the same place (NVME) and I don’t want to do it. I wish to flash the kernel and bootloader into the NOR-flash (nx) and only the rootfs into the NVME.
I didn’t find any specific instructions regarding how to flash just the rootfs into one place and just the kernel into another.

  1. You wrote:

Blockquote
2. You have to know the difference between “boot kernel from external drive” and “mount rootfs from external drive”* Boot initrd/kernel from external drive means the uboot (Nano/TX2 series) or cboot (NX/AGX) will search the kernel and extlinux.conf from your external drive according to the boot order. Once it finds the kernel and extlinux.conf, it will use the info from it to mount the file system. * Mount the file system means kernel will mount the file system according to your “root=” in kernel command line.
For example, the most common usage is root=/dev/mmcblk0p1 which will boot from sdcard (nx/nano) and emmc.
→ Thus, if you just want to increase your disk space, you don’t really need to boot kernel from external drive, you just make sure the file system is mounted from external drive.
→ It means there are lots of variety that you can use. For example, you can boot kernel from usb and mount the file system from NVMe.

So from my understanding , I have to edit the extlinux.conf file with the “root=” arg and point into the NVME that flashed with my rootfs.

2.1. How I now what to write in this argument in order for the kernel to recognise the correct NVME device.

2.2. The extlinux.conf file resides inside the rootfs. the kernel has to read it in order to find the rootfs. So I got this circular condition: find rootfs and read extlinux.conf in order to find rootfs. So I little confused.

Thank you.

Make it more easier to understand…where the extlinux got loaded decides where to load the kernel.

Which means extlinux.conf needs to be same disk as your kernel.

But you could also change that in cboot… this is open source things, so what I said could be changed too.

I would say you could play more with the cboot and check uart log first… it will be more helpful than reading my theory here …

I don’t know what make you feel messy in the uart log. Just focus on the cboot part and you will notice it is just running the scan on each disk to find out the extlinux.conf…

I’ll do it now.
From what you just said, the bootloader is looking for the extlinux.conf , not the kernel. Now it makes sense.

Regardless the cboot (I’ll read its log and play with its settings) -
how can I flash the kernel and the rootfs into different places?
Lets assume I’ll manage to find the setting which direct cboot to load the kernel from specific path (the rootfs/extlinux.conf it searches device by device). I still need to flash the kernel into a seperate device than the one containing the rootfs/extlinux.conf .

for example:
bootloader - NOR flash (thats our only option)
kernel - NOR flash
rootfs - NVME

thanks

I read the developer guide.
Is it possible to flash only APP partition to device A , flash only kernel partition to device B and since there is no LINUX variable in extlinux.conf the cboot will load the kernel from kernel partition?

The actual problem here is what you want to do is never valdated. So even I don’t know whether cboot is able to read kernel from NOR flash.

I am sure it can read usb/nvme/sdcard, but QSPI NOR flash is unknown.

You need to dig into the cboot source.

Ok, I apologize. Maybe I mixed between different things.

In previous comments you corrected me by saying that there is no emmc in xavier nx but a NOR flash.
So basically each time I flash the system.img (contains among others the APP and kernel partitions) into mmcblk0p1 I flash it into the NOR flash and cboot loads the kernel partition from there.

Why its different now?

Hi,

There is a misunderstanding here. Let me explain this again.

We have 2 kinds of modules. SDcard module and emmc module.

Sdcard module is formed by QSPI-NOR flash + your sdcard. → This is for devkit

Emmc module is also QSPI-NOR + emmc. → This is for production module/ custom board case

Bootloader are all on the QSPI-NOR. And APP partition (system.img) is on the emmc or sdcard.

BTW, above cases are only for NX. If you change to TX2-NX, AGX or nano, then it would be different.

Ok,
So my case is the Emmc module.

I want to put APP partition in NVME (using initrd flash
and put kernel partition in Emmc (using flash.sh)
Then , according to the dev guide , the cboot will search for the APP partition in all devices, will find extlinux.conf and since there is no LINUX var in it , it will load the kernel from the kernel partition.

Is that possible?

Hi,

  1. Your case is possible. But the problem we have now is initrd flash cannot work on your board, as our discussion of another thread.

  2. You just need to modify extlinux.conf on emmc and add root=/dev/__your_nvme_node. When cboot scans out the extlinux.conf from kernel, it will know where to mount the rootfs. And that will be your nvme drive.

Thanks,

  1. I’m on it. I do have a solution for that.

  2. If I flash my APP partition into the NVRAM then the extlinux.conf will be in NVRAM and not in emmc. Is that correct?

  3. How I know which value to put in the root=/dev/__ for my NVME? does it have a permanent device name each time I flash the nx module (maybe it can change)

Hi,

For (2) I am not sure about “NVRAM” you mentioned here.

For (3), basically, it should be always same name. Or you can write a udev rule to make sure it will always be the same name.

Actually, even “mmcblk0p1” is similar case… you don’t know your emmc will be mmcblk0p1 but you still flash it with mmcblk0p1.

Thank you for answering,
I think we have a small misunderstanding so I’ll list all the steps , that according to my understanding from your answers , I have to do:

  1. Edit extlinux.conf on my host to specify the root path and point to my NVME(SSD) device.
  2. Flash rootfs only (APP partition) into my NVME device I have in my custom board (using the initrd “method”). The NVME and nx module are connected by PCIe.
  3. Flash rest of partitions (kernel,bootloader and all the rest) into emmc (using flash.sh)
  4. Cboot will find APP partition in my NVME device and the extlinux.conf
  5. extlinux.conf doesnt contain LINUX var , hence cboot will load kernel from kernel partition flashed in the emmc.

Is that correct?

Yes, it sounds correct. So you don’t want to put a rootfs to your emmc which contains a extlinux.conf?

If so, then yes, you can try your steps.