Exact steps to Extend /move file system Root on TX1 to use Vnand m.2 on J120

I have tried this multiple ways and I can create mnt/. one thing though I cannot use gdisk (incorrect architecture faults) I compile my own kernel fine and tried all the ways other users used but eventually causes issues with jet pack components/packages and compilation and cache issues. Its hard for me to ask for advice because I’m limited to the amount on info I can share because of the nature of the project. This embedded architecture is fairly unique and it seems very easy to effect its optimization- Plus cant believe out of all things storage is being this difficult!!

Long story short I have…

JetPack-L4T-2.3
ROS kinetic

Auvidea J120 rev2 carrier
Samsung NVMe 950 pro 512GB PCIe

I just need file system Root on the vnand and work with other blocks correctly. If anyone needs more specifics let me know

Which gdisk are you using…on Jetson, or on desktop host? If Jetson, which version of L4T (“head -n 1 /etc/nv_tegra_release”)?

Is your goal to place the operating system (root partition) on the NVMe storage?

When NVMe is connected to the Jetson, what is the output of “lsblk”?

Yes that is my goal to place the operating system (root partition) on the NVMe storage. I am Using ROS/Cafe/opencv/EKFslam/RTAB for localization/navigation/target tracking. Using stored aerial/target images with no GPS. This is why I need storage.

ubuntu@tegra-ubuntu:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0rpmb 179:16 0 4M 0 disk
mmcblk0 179:0 0 14.7G 0 disk
├─mmcblk0p1 179:1 0 14G 0 part /
├─mmcblk0p2 179:2 0 2M 0 part
├─mmcblk0p3 179:3 0 4M 0 part
├─mmcblk0p4 179:4 0 2M 0 part
├─mmcblk0p5 179:5 0 6M 0 part
├─mmcblk0p6 179:6 0 4M 0 part
├─mmcblk0p7 179:7 0 6M 0 part
├─mmcblk0p8 179:8 0 2M 0 part
├─mmcblk0p9 179:9 0 2M 0 part
├─mmcblk0p10 179:10 0 20M 0 part
├─mmcblk0p11 179:11 0 64M 0 part
├─mmcblk0p12 179:12 0 64M 0 part
├─mmcblk0p13 179:13 0 4M 0 part
├─mmcblk0p14 179:14 0 2M 0 part
├─mmcblk0p15 179:15 0 6M 0 part
├─mmcblk0p16 259:0 0 6M 0 part
├─mmcblk0p17 259:1 0 2M 0 part
└─mmcblk0p18 259:2 0 496M 0 part
nvme0n1 251:0 0 477G 0 disk
└─nvme0n1p1 251:1 0 477G 0 part

This is current

ubuntu@tegra-ubuntu:~$ head -n 1 /etc/nv_tegra_release

R24 (release), REVISION: 2.1, GCID: 7791156, BOARD: t210ref, EABI: aarch64, DATE: Thu Sep 29 00:59:21 UTC 2016

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0rpmb 179:16 0 4M 0 disk
mmcblk0 179:0 0 14.7G 0 disk
├─mmcblk0p1 179:1 0 14G 0 part /
├─mmcblk0p2 179:2 0 2M 0 part
├─mmcblk0p3 179:3 0 4M 0 part
├─mmcblk0p4 179:4 0 2M 0 part
├─mmcblk0p5 179:5 0 6M 0 part
├─mmcblk0p6 179:6 0 4M 0 part
├─mmcblk0p7 179:7 0 6M 0 part
├─mmcblk0p8 179:8 0 2M 0 part
├─mmcblk0p9 179:9 0 2M 0 part
├─mmcblk0p10 179:10 0 20M 0 part
├─mmcblk0p11 179:11 0 64M 0 part
├─mmcblk0p12 179:12 0 64M 0 part
├─mmcblk0p13 179:13 0 4M 0 part
├─mmcblk0p14 179:14 0 2M 0 part
├─mmcblk0p15 179:15 0 6M 0 part
├─mmcblk0p16 259:0 0 6M 0 part
├─mmcblk0p17 259:1 0 2M 0 part
└─mmcblk0p18 259:2 0 496M 0 part
nvme0n1 251:0 0 477G 0 disk
└─nvme0n1p1 251:1 0 477G 0 part /mnt/1df8ed93-04eb-4746-a822-6f775fe181f5

Sorry I meant to say Gparted not gdisk. Gparted is what I cant install

I realize I am PCIE m.2. That is part of my question does Jetson still see it as SCSI ?

I have not used M.2, but there is no reason I can think of that it wouldn’t be the same as any other SATA drive (other than the name of the device special file…there is a conversion between SCSI and ATA which should make the M.2 the same as SATA). Jetson will treat this the same as any other Linux distribution, so as long as you know that device special file name all standard disk commands and operations should work.

You should be able to use “lsblk /dev/nvme0”.

I’m really surprised if gdisk or gparted fail, especially if they show any kind of invalid architecture error (I can use both on a JTX1 using R24.2). There are some hiccups on installing some programs, but this is easy to work around.

What is the error output from these (knowing why this fails might offer clues on why other things fail as well):

sudo gdisk /dev/nvme0
sudo gparted /dev/nvme0

this is what I have now,

ubuntu@tegra-ubuntu:~$ sudo gdisk /dev/nvme0n1p1
[sudo] password for ubuntu:
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present

Creating new GPT entries.

Command (? for help): sudo gparted /dev/nvme0n1p1
You may need to edit /etc/fstab and/or your boot loader configuration!

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): sudo gparted /dev/nvme0n1
You may need to edit /etc/fstab and/or your boot loader configuration!

////////////////////////

Partition number (1-2): 1
Partition GUID code: 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem)
Partition unique GUID: 49568AAC-FC8C-4960-BBD8-DF501B1F94E5
First sector: 34 (at 17.0 KiB)
Last sector: 2047 (at 1023.5 KiB)
Partition size: 2014 sectors (1007.0 KiB)
Attribute flags: 0000000000000000
Partition name: ‘Linux filesystem’

Command (? for help): i
Partition number (1-2): 2
Partition GUID code: 0657FD6D-A4AB-43C4-84E5-0933C84B4F4F (Linux swap)
Partition unique GUID: 56C2B723-F9CF-4E7C-B2D2-6EA9BE330822
First sector: 2048 (at 1024.0 KiB)
Last sector: 1000215182 (at 476.9 GiB)
Partition size: 1000213135 sectors (476.9 GiB)
Attribute flags: 0000000000000001
Partition name: ‘SSDnand’

So now how do I move the root filesystem to the new drive but leave “/boot” on eMMC like you suggested in the other thread? And for boot gedit /extlinux file like this

MENU TITLE p2371-2180 eMMC boot options

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
FDT /boot/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
APPEND fbcon=map:0 console=tty0 console=ttyS0,115200n8 androidboot.modem=none androidboot.serialno=P2180A00P00940c003fd androidboot.security=non-secure tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec={lp0_vec} nvdumper_reserved={nvdumper_reserved} core_edp_mv=1125 core_edp_ma=4000 gpt android.kerneltype=normal androidboot.touch_vendor_id=0 androidboot.touch_panel_id=63 androidboot.touch_feature=0 androidboot.bootreason=pmc:software_reset,pmic:0x0 net.ifnames=0 root=/dev/nvme0n1p1 rw rootwait

One thing I’d like to know is the complete layout of the partitions on that disk. What is the output from “ls /dev/nvme*”?

Your suggested extlinux.conf would work if “/dev/nvme0n1p1” is an ext4 formatted Linux file system created from one of these combinations:

  1. Sample rootfs plus apply_binaries.sh with proper option
  2. A proper recursive copy of the existing file system.
  3. A clone of the existing file system copied to the SSD.

The naming of a new “root=” leaves the boot files used for configuration on the eMMC and does not switch to the SSD for root file system until after the kernel takes over. Once the kernel takes over it is pure Linux and no longer u-boot; “/boot” becomes irrelevant after that, and so although there would be a “/boot” from the SSD showing up, that particular incarnation of “/boot” would be inert. Things should just work with this edit of extlinux.conf regardless of how you choose to create your new root file system.

Method 1, rootfs+apply_binaries.sh:

For sample rootfs plus apply_binaries.sh you can just mount the drive on the Jetson or on the host, copy the “Tegra_Linux_Sample-Root-Filesystem_R24.2.0_aarch64.tbz2” (or whichever version you use) to the SSD, and run:

sudo tar xjfp Tegra_Linux_Sample-Root-Filesystem_R24.2.0_aarch64.tbz2
rm Tegra_Linux_Sample-Root-Filesystem_R24.2.0_aarch64.tbz2

Wherever you have the driver package installed there will be “apply_binaries.sh”. The trick is it has to be on the same machine as the SSD. You can do this from the Jetson if you copy the driver package there (after that you can delete the driver package). Assuming the SSD is mounted on “/mnt” and that you unpacked the driver package there:

cd /mnt/Linux_for_Tegra
sudo ./apply_binaries.sh -r /mnt
cd
sudo umount /mnt

With the extlinux.conf edited, reboot.

Method 2, Recursive Copy Preserving Permissions:

This can actually be trickier than it sounds. You need to copy real files only, not pseudo files ("/sys", “/proc”, many temp files, and “/dev” are pseudo files). This can also be very slow because you are copying one file at a time. I won’t give the details on this since there are so many ways to do this (or to do this wrong), but one way is to use rsync while telling it to not cross file system boundaries and to preserve everything. Another is with a combination of a series of “find” statements, also telling it to not cross boundaries, using option “-print0”, and piping it to xarg with it using option “-0”, and this in turn feeding cp.

Method 3, Using Clones:

This takes longer than rootfs+apply_binaries.sh, but will give you a perfect copy of your existing file system for safe keeping on your host. Advantage…safe copy. Disadvantage…lots of disk space and time required.

To clone the root partition, see this:
http://elinux.org/Jetson/TX1_Cloning
https://devtalk.nvidia.com/default/topic/898999/jetson-tx1/tx1-r23-1-new-flash-structure-how-to-clone-/post/4784149/#4784149

The clone could be loopback mounted and used to manually copy to the SSD, but since it is already clean of any kind of pseudo files, it’s very straightforward. Using sudo a recursive copy with something like “sudo cp -adpR” from the clone to the SSD partition would do everything you need. The clone essentially filters out all of those files you’d have to go through efforts to avoid from a running system.

The easiest method is just create a new file system and apply_binaries.sh. If you want to keep what you have already set up, then I recommend the clone and copy onto the SSD. If you are adept with find, xargs, and cp with preserved permissions, and want to avoid the clone while still needing to preserve the original file system, you could do it the hard way (or the slightly easier variation using rsync which is a bit safer than step-by-step find/xargs/cp).

So I just git back on it and still cant get system to boot using Method 1.

this is my output of “ls /dev/nvme*”

ubuntu@tegra-ubuntu:~ ls /dev/nvme* /dev/nvme0 /dev/nvme0n1 ubuntu@tegra-ubuntu:~

What is the output of:

sudo gdisk -l /dev/nvme0

as of right now-

GPT fdisk (gdisk) version 1.0.1

Usage: gdisk [-l] device_file

ubuntu@tegra-ubuntu:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0rpmb 179:16 0 4M 0 disk
mmcblk0 179:0 0 14.7G 0 disk
├─mmcblk0p1 179:1 0 14G 0 part /
├─mmcblk0p2 179:2 0 2M 0 part
├─mmcblk0p3 179:3 0 4M 0 part
├─mmcblk0p4 179:4 0 2M 0 part
├─mmcblk0p5 179:5 0 6M 0 part
├─mmcblk0p6 179:6 0 4M 0 part
├─mmcblk0p7 179:7 0 6M 0 part
├─mmcblk0p8 179:8 0 2M 0 part
├─mmcblk0p9 179:9 0 2M 0 part
├─mmcblk0p10 179:10 0 20M 0 part
├─mmcblk0p11 179:11 0 64M 0 part
├─mmcblk0p12 179:12 0 64M 0 part
├─mmcblk0p13 179:13 0 4M 0 part
├─mmcblk0p14 179:14 0 2M 0 part
├─mmcblk0p15 179:15 0 6M 0 part
├─mmcblk0p16 259:0 0 6M 0 part
├─mmcblk0p17 259:1 0 2M 0 part
└─mmcblk0p18 259:2 0 496M 0 part
nvme0n1 251:0 0 477G 0 disk /mnt/74a29476-292f-48ec-a82f-1fb57708caf7

ubuntu@tegra-ubuntu:~$ sudo gdisk -1 /dev/nvme0
[sudo] password for ubuntu:
GPT fdisk (gdisk) version 1.0.1

Usage: gdisk [-l] device_file

When ever I go through the process of the Example 1 this is what I get could it be something with the custom kernel I had to compile on the mmcblk0 to use the m.2 vnand?

What it looks like is the nvme0 does not have any partitions, at least nothing it recognizes. It seems like somewhere prior to this you had created nvme0n1 as a partition on nvme0…is there any reason why that partition should have disappeared, e.g., deleting it or using a different M.2 device? I’m wondering about your custom kernel compile, but if gdisk can see nvme0 then I tend to think that this is not the problem; since the kernel can see the disk as a whole an unreachable partition within the disk is unlikely to be a fault of any kernel driver for the M.2 so long as the partition is GPT or MBR partition type. gdisk can understand both MBR and GPT style partitions.

What happens if you use sudo on the Jetson and try to create a single partition on nvme0 with partition type “8300” (this is a standard “Linux” file system partition label…I see “8305” is listed as an ARM64 root partition type, but it is questionable if u-boot can work with that and perhaps there is no reason to test)?

Hi linuxdev,
I followed method 3 you suggested previously. What is the next step I must take after using sudo cp --adpR to put the clone onto the SSD ?

After coping the image to the ssd and rebooting the TX1 I got the following error:

[   12.447255] EXT4-fs (nvme0n1): VFS: Can't find ext4 filesystem
Press Enter for maintenance
(or press Control-D to continue):

Thanks,
Callum

Rsync is perhaps a better choice since it may have a better understanding of files, but in a pinch the cp should work too (assuming you loopback mounted the clone and the source is the clone and destination is ext4 formatted).

Basically it sounds like you tried to copy files rather than a partition, and that the destination was a partition without ext4 formatting. What was your source and destination? Loopback mounted clone and an existing ext4 formatted partition? Or just the disk as a whole?

Note that when you clone you get both a larger “.raw.img” file, and a smaller “.img” file. Only the flash software seems to understand the sparse image format of the smaller file, but the larger “.raw.img” is a bit-for-bit exact copy of the partition. If you happen to know the exact starting byte on your alternate disk where the first partition starts (first regular partition), then dd can copy this…and the file system will exist with it. You might have to use gdisk to fix any mismatched partition extents after that if this partition is not the exact size of the partition being overwritten with dd.