TX1 flash freezes at "Making system.img" with JetPack 2.3

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

My TX1 flash freezes at “Making system.img” step with JetPack 2.3. Any idea on what might cause this?

By “Freezing” I mean nothing happens for 3 hours after “Making system.img…” is displayed. I have read that making the system.img can take a while but I am not sure how long to expect.

Here is the flash log:

Warning: missing eksfile (bootloader/eks.img), continue…
copying
bctfile(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/bootloader/t210ref/BCT/P2180_A00_LP4_DSC_204Mhz.cfg)… done.
copying
initrd(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/bootloader/l4t_initrd.img)… done.
populating kernel to rootfs… done.
populating initrd to rootfs… done.
populating extlinux.conf.emmc to rootfs… done.
populating
/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/kernel/dtb/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb to rootfs… done.
done.
copying
bcffile(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/bootloader/t210ref/cfg/board_config_p2597-devkit.xml)… done.
Existing
sosfile(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/bootloader/nvtb
oot_recovery.bin) reused.
copying
tegraboot(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/bootloader/t2
10ref/nvtboot.bin)… done.
Existing
bpffile(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/bootloader/bpmp.bin) reused.
copying
wb0boot(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/bootloader/t210ref/warmboot.bin)… done.
Existing
tosfile(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/bootloader/tos.img) reused.
copying
dtbfile(/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1/kernel/dtb/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb)… done.
Making system.img…

Other info:

  1. On my first TX1 flash attempt, I had a different error message:

Making system.img…
./flash.sh: line 466: 3240 Killed mkfs -t $4 “${loop_dev}” >
/dev/null 2>&1
formating ext4 filesystem on system.img failed.

  1. My host PC is running Ubuntu 14.04 successfully from a 64Gb MicroSD card on via a USB 3.0 hub.

Thanks in advance.

Jon

2

Is the SD card formatted as ext4? If it is the default VFAT of most flash cards it won’t work…it’ll look sort of like it works, but in the end it will fail. Make sure the sample rootfs itself was unpacked to ext4, not VFAT.

One of the details at that stage which may be subtle at times is that you need a lot of disk space…15GB for the sparse image, and probably another 3 for the compressed version. This is in addition to everything else required from hard drive during the flash (the sample rootfs itself is about the same size as the compressed/sparse system.img). Make sure your destination partition has many spare GB.

Creating and formatting the 15GB of file space is normally slow, but I suspect on a flash card (if it works), that the speed is much slower yet. 3 hours might even be reasonable. You can occasionally check the size of “bootloader/system.img” and “bootloader/system.img.raw” to see if either is growing…if so, it’s still going and doing as it should. system.img is created as an uncompressed file of the exact size that the partition will be, then moved to the name “system.img.raw”, and this in turn is used to create a compressed version as “system.img”. FYI, even USB2 is far faster than many of the flash cards. Just check those file sizes on occasion once system.img creation is started to see if any of it is growing.

3

Thanks for the help!

  1. As far as I can tell the SD is formatted as ext4, for the relevant partitions (dev/nvme0n1p1 is the exception):

                                                 Disk Drive: /dev/sda
                                       Size: 64087916544 bytes, 64.0 GB
                             Heads: 255   Sectors per Track: 63   Cylinders: 7791

Name Part Type FS Type Directory Size (MB) Used

                  Pri/Log        Free Space                            16.78       *

dev/sda1 Primary ext4 / 32000.00 53%
dev/nvme0n1p1 Boot? vfat /boot/efi 268.00 11%
Pri/Log Free Space 0.45 *
dev/sda2 Primary ext4 /home 29999.76 33%
sda3 Primary swap 1999.64 *
Pri/Log Free Space 71.31 *

A rootfs directory, with files and subfolders, exists in:
/home/jdq/JetPack/64_TX1/Linux_for_Tegra_64_tx1, which is ext4

  1. I found “bootloader/system.img” and the properties indicate it is 15Gb in size (while “Making system.img” is still executing). “bootloader/system.img.raw” did not exist, at time of writing this reply.

  2. I am running Ubuntu on an i7 Surface Pro 4, via the MicroSDXC rated at 150 MbpS read (UHS-II, U3, Class 10 in a USB 3.0 adapter and hub). It’s pretty fast for most things.

4

At some particular byte size very close to 15GB system.img gets renamed to system.img.raw. From this a new system.img is created (the raw file is uncompressed, the new version is compressed). So it is very very close to creating system.img.raw. What is the exact size you named in the flash? Maximum size is “14580MiB”, which is an exact raw image byte size of 15288238080 bytes (1458010241024). If “14GiB”, then the exact size at the moment of switching to making the compressed version would be 15032385536 bytes (1410241024*1024).

Watch if system.img reaches an exact size, or just increases some. Once size increase stops, expect system.img.raw to be created via rename of system.img. A short moment after that system.img will begin to be created again in its new form.

Btw, you can see a file system type via “df -T”.

5

bootloader/system.img was exactly 15032385536 bytes (1410241024*1024) at 15:10 Sat (last modified datetime) and 6 hours later it had not changed. It was never renamed bootloader/system.img.raw.

I deleted bootloader/system.img and restarted the JetPack install again. This is what happened:

  1. I got this message about 1 second after "Making system.img… ":

Making system.img…
./flash.sh: line 466: 3240 Killed mkfs -t $4 “${loop_dev}” >
/dev/null 2>&1
formating ext4 filesystem on system.img failed.

  1. I checked bootloader/system.img and it existed with a size of exactly 15032385536 bytes (1410241024*1024), datetime was Sun 12:10 (about the time I got the above error). There was definitely not enough time to create a real 15Gb file.

  2. I was prompted to check the log files and then hit [Enter]

  3. When I hit [Enter] I was prompted to press the “Reset” button on the TX1 board so that the “GUI would display”. (I assume it was expecting the TX1 to boot).

  4. Then the the install program started looking for an IP Address and hung.

  5. I cancelled the install at that point.

The “formating ext4 filesystem on system.img failed.” error seems to be the issue but I don’t know why.

6

The initial 15GB image is created without need for loopback. Formatting requires loopback. What you’ve described is loopback failing. It’s quite possible flash continued on with 15GB of NULL bytes from an unformatted image. I do not believe the flash program differentiates between 15GB of NULL bytes and 15GB of properly formatted image. The boot loader itself would still work.

So, debugging loopback is required. If your host’s Linux kernel does not support loopback this would fail…but so far as I know there isn’t a Linux distribution anywhere which does not support this by default. If root or root authority (sudo) is not used, then you cannot create a loopback device. If flash is run on command line you’d be required to use sudo or root login. In the case of JetPack I’m not sure what the setup is, but I think you run it as non-root and then it prompts for password to use sudo…someone knowing JetPack would have to comment on this.

Loopback devices are created as numeric “loop#” in “/dev”, e.g., “/dev/loop0”, “/dev/loop1”, so on. The command “losetup -f”, when run as anyone other than root, simply lists the name of the first unused loopback device…but if that device does not yet exist, then the device will not be created. Running “losetup -f” as root will not only name the first unused device, this will also create the device if it doesn’t exist.

Loopback devices are virtual and don’t really exist on the file system. There may be intentional limits on how many can be created to avoid consuming too much of some resource. If you had a program with a defect which endlessly creates loopback devices without destroying them, then there would be no free loopback devices and new devices would be refused (even for root). Check “ls /dev/loop*” and see how many loopback devices there are. See if “losetup -f” names a device which exists in “/dev”, versus if it names a device which still needs creating.

7

I switched the host PC for the JetPack 2.3 install/flash to one that was running Ubuntu from a USB 3.0 SSD drive and the flash of the TX1 worked fine.

On the host PC that worked, I noticed that the JetPack 2.3 install directory was 28Gb in size once the install and flash completed successfully.

I am wondering if its /home partition did not have enough space to complete the flash process? The reported disk space available was:

Name Part Type FS Type Directory Size (MB) Used

dev/sda2 Primary ext4 /home 29999.76 33%

This suggests that there was only 20Gb available on the /home partition when the JetPack flash appears to need about 28Gb. I am sure the virtual loopback devices confuse things somewhat.

Would the error I got be generated if space was insufficient?:

Making system.img…
./flash.sh: line 466: 3240 Killed mkfs -t $4 “${loop_dev}” >
/dev/null 2>&1
formating ext4 filesystem on system.img failed.

I would have expected a less misleading message if it was a disk space issue. Maybe the JetPack flash process is not pre-checking space availability and the system.img format just fails due to lack of space?

I thought I should get some feedback on this theory since to test it I will have to re-size my host PC’s /home partition and may in fact discover that a 64Gb microSD card is not big enough for JetPack install and everything else I use it for.

Thanks

8

Without enough space you would definitely get some truncated files and unexpected results. Often there are temp files created during tasks like compressing or extracting files, so the final result is probably less than what was required at peak while the process ran. This would not effect loopback directly (loopback is a driver in RAM covering a file, but not a file in itself), but there is a strong chance that any kind of lock files used with the loop device would not be able to be created and would break loopback.

The disk space issue isn’t particularly a JetPack issue, that’s just the way running out of disk space on Linux is handled. You can tweak resource limit settings and change this to a more desirable behavior, but few people do this. JetPack itself could check available disk space and if the file system type of the rootfs directory is native Linux, but almost any application out there depends on the user having a sufficient environment rather than checking on and enforcing such things (in other words it would be a nice feature, but not a typical feature).

9

Thanks for your help. I will try the JetPack 2.3 install/flash again with more free space and see what happens. However, I think it would be a good idea to let people know that a minimum of 30GB of free space is required to flash the TX1 using JetPack 2.3. While it makes sense in hindsight, I don’t think one would guess that 30Gb free space would be required.