Flashing with JetPack 3.2.1 fails - "Read-only file system"

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hello, I am using JetPack 3.2.1 to flash the TX2 with an Orbitty carrier board. The TX2 has 28.1 on it and I would like 28.2.

I installed JetPack on my host but flashing to the device fails. When following the NVIDIA instructions here, at step 14 a terminal appears and tells me to put the TX2 into force recovery mode. I do and it proceeds to print a long list of directories and files in this format:

Making system.img... 
rm: cannot remove 'mnt/README.txt': Read-only file system
rm: cannot remove 'mnt/home/nvidia/examples.desktop': Read-only file system
rm: cannot remove 'mnt/home/nvidia/.bashrc': Read-only file system
rm: cannot remove 'mnt/home/nvidia/report_ip_to_host.sh': Read-only file system
rm: cannot remove 'mnt/home/nvidia/.profile': Read-only file system
rm: cannot remove 'mnt/home/nvidia/.bash_logout': Read-only file system
rm: cannot remove 'mnt/home/ubuntu/Downloads': Read-only file system
rm: cannot remove 'mnt/home/ubuntu/.gnupg/private-keys-v1.d': Read-only file system

… …

rm: cannot remove 'mnt/etc/NetworkManager/VPN/nm-pptp-service.name': Read-only file system
rm: cannot remove 'mnt/etc/NetworkManager/conf.d/10-ubuntu-fan.conf': Read-only file system
rm: cannot remove 'mnt/etc/NetworkManager/system-connections': Read-only file system
rm: cannot remove 'mnt/etc/NetworkManager/dnsmasq.d': Read-only file system
rm: cannot remove 'mnt/etc/passwd': Read-only file system
rm: cannot remove 'mnt/etc/nsswitch.conf': Read-only file system
rm: cannot remove 'mnt/media': Read-only file system
rm: cannot remove 'mnt/mnt': Read-only file system
clearing ext4 mount point failed.

CTI’s instructions for the Orbitty board (see “CTI-L4T BSPs for TX2 and TX2i L4T V28.2 and Higher” section) result in the same output. I believe NVIDIA’s installer runs the same flash.sh script that CTI’s instructions tell you to run from the command line.

Inside the flash.sh script I believe these errors are coming from the

build_fsmsg()

function but I’m not exactly sure what it is doing. Googling has been no help.

How can I flash 28.2 to my TX2? Thank you.

2

What location is JetPack being run from on the host? I’m guessing you are running JetPack from perhaps some temporary media location. I’m trying to narrow down where on the host PC the “mnt/” directory is which it is referring to since it is a relative path. If this is some sort of alternate or temporary file system (e.g., an SD card or USB disk), can you describe the storage device?

3

JetPack is installed on a larger HDD in my host, the linux OS is stored on a SSD. Should I try installing and running JetPack from the SSD? I’ve been fiddling with permissions but the HDD is permanently installed in my computer and I’ve never had permissions issues with it before.

4

It might help to understand part of the normal flash process where this error is occurring. I don’t know yet what the error source is, but this is basically describing what needs to be tested. Keep in mind that if I mention JetPack I’m really talking about the driver package for the flash side since JetPack is just a GUI to this.

First the driver package is unpacked. This creates the “Linux_for_Tegra/” directory and all of its content, with the exception that “Linux_for_Tegra/rootfs/” is empty (the “REAMDE” file doesn’t count). The content of “rootfs/” is from unpacking the sample root file system tar package. At this point the “rootfs/” is purely Ubuntu without any special drivers for Tegra. Running “apply_binaries.sh” is copying the drivers for Tegra into “rootfs/” (the tar package is from the driver package content of “Linux_for_Tegra/nv_tegra/nvidia_drivers.tbz2”). Once those drivers are unpacked into “rootfs/” you truly have Linux for Tegra (“L4T”).

The driver package itself is just a simple user space program which is not unpacked with any kind of special privileges. However, the sample rootfs unpack would be completely wrong if it isn’t unpacked with sudo (or root) permissions because files have to have permissions and ownership preserved (a regular user can’t do that). The same is true for the “apply_binaries.sh” step…root or sudo must be used to run this command because the files being unpacked overwrite other root owned files in “rootfs/”. You have a permissions problem, and so some combination of unpacking permission issues for sample rootfs and apply_binaries.sh has occurred.

Before you go and check your steps, consider first that if the “rootfs/” directory exists on some file system which does not natively support all Linux permissions, then that entire partition is incapable of taking advantage of sudo. NTFS for example has no concept of a group, and there is absolutely no possibility these non-Linux file system types can succeed.

Another possible issue is that this is a lot of data. The content of “rootfs/” is the entire operating system going into the Jetson. This isn’t as big as the full partition, but it is several GB. If you run out of disk space while unpacking the sample rootfs or if you run apply_binaries.sh against a directory which is full, then it won’t matter that the write is prevented via lack of permission or if the denial is because the disk is just full has no room left (though it is true the error messages would differ).

We could go on and look at whether you used sudo or not, but the error message is a bit more unique and descriptive so we don’t even have to look at sudo (at least not at this stage). To quote that error:

Read-only file system

Not even root is allowed to write to a file system which is read-only. Your SSD which the operating system is on is obviously read-write, and this isn’t an issue. However, the method of mounting the disk partition for which “rootfs/” is on is in question. Read is failing because there is something non-standard about this partition. Something in the mounting of this disk, or in the type of file system of this disk, is forcing read-only.

You will want to describe how the disk is mounted, e.g., if it is automounted or if you set it up manually. Perhaps the disk is mounted on every boot, and if so, you might describe the line for that disk in the “/etc/fstab” file.

Or, the file system type the partition is formatted with might itself be the issue. The command “df -H -T /where/ever/you/look” will tell you both the spare disk space of the disk at the point named, and also this will tell you the file system type. I am guessing that either the type is not native Linux (e.g., it isn’t ext4), or else the mount options were incorrect to be able to write to this disk. What do you get from “df -H -T /where/ever/it/is/Linux_for_Tegra/rootfs/”? If this is ext4, then we have to consider how the disk was mounted.

5

Thank you for this detailed reply, I think you identified the problem. The HDD I unpacked JetPack on is formatted with NTFS since my system is dual booted (linux and Windows). I didn’t realize this could cause problems, I’ve been using it with this format for almost a year now and can read and write files on the HDD through the linux CLI and GUI. However, I have not confirmed that switching away from NTFS allows me to flash since a coworker has successfully flashed from their machine and the project has proceeded, and I haven’t reformatted my drive yet.

What disk formats should be compatible with the flashing process? You mentioned ext4 but since my system shares disks between OSs it would be nice if it worked for exFAT as well.

The reason its on the HDD in the first place is that there wasn’t enough space available on the SSD that is shared with the OS. Perhaps the real solution is just to buy a bigger SSD.

6

Systems flashed from NTFS will appear to work at first. There will be severe issues though during operation, including that “sudo” will no longer work (have your coworker see what happens from “sudo ls”)…and many GPU operations could require this. So would package updates. You should warn your coworker that it is a guarantee that if the flash was from NTFS where the “rootfs/” subdirectory was not on a native ext4 style partition the install cannot be valid. This absolutely would need to be flashed again…there is no chance of repairing such an installation.

Now if you run out of disk space, then the rootfs image will be truncated. Where it truncates and what gets left out just depends on which point in the file copies the disk ran out of space on.

For success you need a file system which supports the full range of Linux permissions. No Windows or Mac style file system will work. The FuseFS of a live DVD also will not work. “ext4” is the only recommended file system, but some others would work, e.g., ext2, ext3, ReiserFS, JFS, XFS (which I’ve used in the past), and probably Btrfs.

File systems which are Linux, but which might have some sort of permission translation will not work (or most of the time they won’t, and you will never have a guarantee). The biggest example is to not trust NFS for this. NFS could easily hold the tar files used for an install, but then the unpack would have to go directly to a local ext4 partition.

No “FAT” variant of any kind will work (the install will succeed, and then things will go badly wrong when you need root permissions or special group permissions (e.g., to use the GPU as non-root you must be a member of group “video”…FAT and NTFS do not have any concept of “groups” and cannot propagate such a setting from a non-native Linux file system type).

I would recommend a separate disk for a native Linux install and dual booting. If you have to stick with your current system, then at minimum the partition you put JetPack or any flash software on must be from a native Linux file system type, e.g., a separate disk with ext4 formatting which is mounted as a second disk (e.g., “/home/whoeveruare/JetPack/” could be ext4 while the parent directories are not).

7

Thank you for all of this very detailed and useful information. If/when I try flashing from a properly formatted hard drive on this system I will follow up.