NX can't boot after the mmc drive filled up

My NX running JetPack 4.6.1 was working normally but when I rebooted it is showing the following during boot and then goes to a black screen forever and doesn’t make it into ubuntu. The 16 GB mmc drive on my NX was full when I rebooted. /home is mounted on an ssd but / is mounted on the mmc drive. Does anyone know what is wrong or how to fix this?

Sometimes serial console can still log in (depends on temp file creation ability; serial console has the least requirement). Can you log in via serial console to delete files? ssh will usually fail with less of a filled filesystem, but it is easy to test if ssh login works. Also try a console via CTRL-ALT-F3.

Thanks for the fast reply. Unfortunately, we modified our device tree to disable the debug serial port so that we could use it like a normal UART port to connect to an extra device. I also tried CTRL ALT F3 but it didn’t work, the screen stayed black. Is there anything else we could try before just reflashing?

You can clone, remove some content from the clone on the host PC, and then flash with the same rootfs (which is the clone less the removed files). It takes a long time to clone and flash compared to just flashing, so it depends on how much you want to keep the old content. You’d clone and reflash using the exact same JetPack release (or third party vendor software if this originally created the content, although mostly that change is just the device tree) if you choose to go that route.

There is no method which I know of to do this from bootloader console. If you had some other bootable external media, then in theory you could boot to that and remove files. However, to do this generally requires setting it up ahead of time. Once this happens it is too late in most cases, although someone who works more with external boot media might be able to mention some way to boot to a USB hard drive without first modifying extlinux.conf. I don’t know if that is possible, and I have doubts.

A clone produces both a “raw” partition image, plus a "sparse’ image. They are the same, but the sparse image excludes empty space; when flashing the partition expands to the same size as the raw clone. When flashing you can use either the raw or sparse image so long as it is placed at “Linux_for_Tegra/bootloader/system.img”. The larger raw file takes longer to flash, but this is also the one you can mount and edit on the host PC. You can convert the edited raw file to a sparse image for flashing faster, but this in turn takes time and disk space, so mostly I wouldn’t bother with that unless you are going to flash multiple times with that image.

You have an eMMC model, but I’m not sure of some details. A typical NX clone would go something like this with the Jetson attached in recovery mode:
sudo ./flash.sh -r -k APP -G my_backup.img jetson-xavier-nx-devkit-emmc mmcblk0p1

If you look in your “Linux_for_Tegra/” directory at the “*.conf” files, then you’ll see the various flash targets. My example “jetson-xavier-nx-devkit-emmc” corresponds to a “jetson-xavier-nx-devkit-emmc.conf” file. You’d pick the one which corresponds to your board. Note that many of those files are symbolic links to equivalent files named after various numeric models. The models are in turn a combination of module and carrier boards. These are human readable and you can sometimes pick the right thing for custom setups based on that. Or you can copy one and edit it for a custom setup. I mention this because I don’t know for certain what to clone and flash as the target (if it works, then you should be ok; clone won’t care about release as long as it detects the recovery board Jetson; reflash with the clone does depend on both the target being correct and the JetPack release matching what created the image in the first place).

If you need help with loopback mount and edit of “my_backup.img.raw” (which is produced at the same time as “my_backup.img”), then just ask.

Thanks.

We ended up putting the NX on a different carrier board without the ssd that /home is mounted on and when we tried booting it then, it went into “emergency mode” and we were able to delete some files. Then when we put the NX back on the original carrier board with the SSD it booted normally.