Tx2 wont boot cause it is out of memory

Hey I was running a program on the jetson tx2 and it created a file that probably used up the remaining memory on the board, Now the jetson cannot boot and is stuck in a boot loop (I cannot log in through ssh as well). I went through this issue Memory issue in Jetson TX2 and it recommends to log in through the serial console but can I do that without the USB-to-TTL Serial Cable? or do I need to order one? Ultimately I am trying to get the jetson to boot and then clone the internal storage onto an SSD and use it instead. Any help you can provide is greatly appreciated and if you can provide step by step instructions to clone to ssd that be great.
Thanks

Most methods of logging in require temp files. The fewest resources required would be via the serial console, which does need the USB-to-TTL serial UART cable (3.3V logic level).

More detail about rescue via clone and restore-from-edited-clone can be found here:
https://forums.developer.nvidia.com/t/jetson-wont-load-up-after-configuring-spi-functionality/112139/4

Clone/restore is a long process, but it isn’t a bad idea to keep this around as a backup in case something does go wrong. Clones can be occasionally refreshed via rsync if desired.

thanks, I’ll go through the link and get the cable ordered. but am I right in assuming that this is because of memory problems?
It displays the screen as in the image below and then reboots and displays the same thing

again in a loop

It could be, but I can’t say for sure. If your content is not valuable, then you could just flash again. If your content is valuable, then you probably want to keep a clone handy anyway, and the loopback mounted raw image clone can be examined and you would know immediately if it is filled. You could correct this on the clone and flash the clone back in.

However, a serial UART cable is so valuable during development you’d probably want this anyway, and so I’d still recommend you get this. If the system is sufficiently low, then you still might not be able to log in to remove excess content, but even so it is worth having that serial console cable around for the future.

I would consider reflashing; to me it doesn’t seem a disk space issue. However, it might be it hypothetically.
As far as i remember disc full issue would cause GUI login loop and could be managed with “Ctrl+Alt+F2”


Here is some more information if it helps in debugging the issue

That information just says “it doesn’t work”, and does not help much with debugging. If the unit really is full and no space left, then you can only repair by removing some content and freeing up space.

The simple method to free space, if you have serial console access, is to delete files over serial console. The nearly-as-simple method is to flash again, but lose current content. The longer method which saves content (which is very time consuming, but not necessarily difficult) is to clone first to a Linux PC, delete content within the clone on the PC, and then flash using the repaired clone instead of generating a default rootfs image.

Note that this is normal behavior for any computer with space filled up, and so if you go to clone from your PC make sure your PC itself does not fill up from the cloning.

I don’t think a program that uses all the system memory (you mean the SDRAM right?) would cause the system cannot boot up.

My suggestion is like other people here… maybe you need to reflash your board.

Also, it would be better to dump the full log instead of taking pictures. The UART log may also tell some useful information.
https://elinux.org/Jetson/General_debug

I could be wrong, but was basing this on eMMC filling up. Partly because of your point that if a program had run and used up RAM, then reboot would fix the issue until the program is run again. @mehulsur , if you can, is it correct that you ran out of space in “/home”?

i am assuming that is the case cause i was low on memory prior to running the program however that might not be the case as i still had about a GB left. I have ordered the UART cable and it is in the mail. once I have that I’ll post a more detailed log.

I’m interested to see where I can learn more about that method please share any links if you can

Does your program generate any output file that would exhaust your disk?

Clone instructions depend on release. Do you know which JetPack/SDK Manager or L4T version you are using?

Hey thanks for the help, i got the Jetson to boot using the UART cable. the issue was a lack of space. I deleted a bunch of model checkpoints and it boots fine normally now. I got a ssd mounted and copied all the filesystem over to it. is the best solution now to change the boot device to the ssd (Are there instructions on how to do this i’m using jetpack 4.3)? Also the jetsonhacks youtube video recommends to change boot options and rebuild kernel and modules but what is the purpose of rebuilding ?

Hi mehulsur,

You can reference Topic-79833 steps to flash image on external devices.

I would personally just mount the external disk to “/usr/local/” after transferring that content there. Booting directly to a disk will be more complicated and likely mean you cannot operate without that. If all build and other content is on “/usr/local/”, then you’ll be able to disconnect the disk and still function, and yet not run out of space. The disadvantage is that booting to the external drive would likely be faster (especially if the external drive is SSD instead of old style hard drive).

If you want information on mounting to “/usr/local/”, then just ask…it is fairly trivial. @carolyuu already pointed at the information for flashing (to run the o/s itself from another partition).

Note: If you use mount to “/usr/local/”, and it isn’t what you want, then you could still use the flash of whole o/s method.