Jetson TK1 - crontab boot deadlock


I’ve made a script to check internet connectivity and comitted it to crontab.
The script would reboot if no ping.
Thing is, apparently at boot time the network is still not ok, so TK1 is in deadlock reboot.
I can’t get access to GUI, neither to command prompt.

I’ve tried keeping shift pressed during boot, also ctrl-alt-f1, to no avail.
Apparently the system has no greeter and no safe mode boot.

Is there anything I could do?

In recovery mode, would I be able to login into L4T and edit crontab in any way?

Thanks in advance.

Serial console will always work unless the system is truly dead. Your other methods of access require drivers and parts of the system to work which are “fragile” under failure conditions…serial console has almost no requirements and usually survives otherwise hopeless conditions. Get a serial console, settings are 115200 8N1.

NOTE: Serial console works even in U-Boot prior to ever loading Linux. If you have alternate boot entries (such as for rescue), then you need serial console to select those entries.

Not sure if I did it right but nothing happened. Installed minicom, configured, but it’s always ‘disconnected’, although I can see the boot status on a monitor connected to the hdmi port. Should I gain any control over the boot steps by using the serial console?

I rebooted in recovery mode, emmc appears to be listed in usb devices as it should.
Now I’m trying to access the file structure, hoping to change boot config somehow, maybe editing the crontab for the specific user, ‘ubuntu’ in this case, what could be tricky I guess.

Thanks for your reply.

It is easy for minicom to get the wrong settings since it was originally designed to talk to a serial port which is a modem (old fashion dial-up). You won’t do any of the “init” functions, e.g., sending “AT” commands. Use speed 115200, 8-bit, 1 parity bit, no stop bits. You can use CTS/RTS (hardware) flow control if you want…it’s optional (I suggest using it if it works for you…not all cables have the flow control pins, and not all software uses it).

One thing about minicom is it hates to use serial terminal device special files not named the way they were named when telephone dial up modems first started being used (e.g., it may like “ttyS0”, but hate “ttyUSB0”…if you use a USB serial UART instead of pure 9-Pin D-sub connectors at both ends you’ll need to use such a name).

Which tty port did you use? Is it a USB serial cable at one end and a 9-pin D-sub at the other end?

An example of minicom on command line (you may need to use sudo if permissions on the tty are not open) with basics:

minicom -b 115200 -8 -D /dev/ttyUSB0

I prefer gtkterm, but have to edit the config file to use a ttyUSB#…I don’t know why this was hard coded in so many serial console programs to not allow selecting a ttyUSB# without editing a file.

If it is set up correctly it should have output almost immediately. Hitting a key at the right point in U-Boot lets you access U-Boot command line (type “boot” to continue boot). Or a bit later (for a very short time) it offers allowing you to pick among your extlinux.conf entries…there is only one by default, so most of the time the step is irrelevant.

Thank you, but that really didn’t do anything. It just shows ‘disconnected’ and nothing happens.
The boot screen shows as it should on a hdmi port connected monitor.

Last Friday I’ve tried to edit the system.img, by exposing it via mount command, as explains.

Downloading, mounting and editing went well.
When restoring, however, the flashing process got stuck at some point, as reported by many other people. Today (more than 48h late), it didn’t complete.
Powering off and on again returned the board to the same (deadlock boot) previous state.

I used this line to restore:

sudo ./ -r -S 14580MiB jetson-tk1 mmcblk0p1

I’ve heard that in more ‘recent’ systems, the -S should be removed.
Honestly I didn’t have a clue how to check that.

Flash from a VM quite often fails with stalling out. Did you try the restore from a VM? A VM is actually expected to fail this way (a slow flash might take about three hours).

I’m still a bit confused about the “disconnected” showing up…am I correct that this is on serial console via minicom? If so, minicom was originally designed to communicate with modems, and this may be minicom trying to initialize and detect a modem (you can skip modem initialization and just use minicom as a terminal). Modems are serial consoles, but they also have some “AT” commands which are sent for initializing a modem and connecting over a telephone to somewhere else…it sounds like it is trying to use the serial UART as a telephone modem and thinks the phone call didn’t succeed.

If possible can you try gtkterm? This was not designed to use a modem, it’s pure serial UART console communications. You can run it correctly for your serial console with the following example (adjust the UART name to your situation, I’m assuming “/dev/ttyUSB0” only as an example):

gtkterm -p /dev/ttyUSB0 -s 115200 -b 8 -t 1

Not a VM, but Ubuntu on ‘safe mode’ with networking, because the system could not boot through the GUI.

Yes, it’s serial console via minicom. Also tried within Windows, no cookie.

I was using a USB to DB9 cable, with PL2303 IC, not FTDI, which some users point to be a failure reason.
It was recognized with lsusb though.

I can try using gtkterm next time I have the system at hand. We were on a tight schedule and I had to
redeploy it to the point of use (a utility) after I got it working again.

After some glitches, I eventually was able to download the system.img, mounting it and
correct the culprit crontab file,
and recover it to the emmc. Took a while, I’d give you that, but it worked, and I was relieved… gotta tell
you that… I was on the verge of despair indeed…
The steps I took were quite general ones, meaning they could be used to repair virtually any system
still capable of forced recovery booting.

Nonetheless, thanks once more for your time and input. Much appreciated.

Dang, I like cookies…I hate it when there are none!

This is somewhat of a long shot, but does your Linux motherboard support UEFI and GPT partitions, but your hard drive with Linux is on an old style BIOS partition? Typically “sudo fdisk -l” shows up normally for any “/dev/sd*” with old style BIOS partitions, but “sudo gdisk -l /dev/sd*” would complain about no valid GPT partition if this is the case. GPT does not use the old primary/extended/logical partition scheme…this was a limitation on implementing larger hard drive support. Having a primary/extended/logical partition has no meaning under GPT.

If you are mixing old style BIOS partitions with a motherboard capable of UEFI (even if the motherboard is in BIOS compatibility mode) it is possible you are running into an Ubuntu installer bug which thinks you are going to have UEFI support for some drivers…but the support doesn’t exist in a BIOS partition scheme, which in turn would mean some device drivers won’t work (the installer has no way to force pure non-UEFI install). This would be very likely to cause an invalid flash if it is the case…but like I said, the odds are low you actually have this setup.

The gist of the Ubuntu bug is that part of the installer may realize you are using old style BIOS partitions, and other parts may think that since the motherboard supports UEFI that you must have UEFI partitions…even if you are in BIOS compatibility mode. Ubuntu seems unable to install correctly in mixed environments such as this. Should this be the case you may need to use a non-Ubuntu distribution which better handles BIOS partitions on a UEFI-capable motherboard. Oddly, the live DVD will work…unfortunately the live DVD doesn’t support everything needed to flash, e.g., loopback support.

If you flash on command line then Fedora works quite well.

Thank you for your thorough explanation.

Well, at the end the system.img download/edit/recover process did the trick, and the system is working 5/5 since then.
Not sure if it would be fair to mark this turnaround as a technical answer but, hey…
whatever keeps the wheel turning, right?..

(duplicated, server had problems…)