Sometimes the xavier takes about 5 minutes to boot up. We caught the error with a monitor plugged in at one point and it had some output on the screen about a kernel panic. It seems like it waits 5 minutes at this screen and then will reboot and it works. It seems like this doesn’t get written to a log anywhere and I’m having trouble tracking the problem when I don’t have a monitor hooked up to it. Is there any techniques to debug this issue?
The filesystem has too many errors (the journal cannot adjust for that much error). There are a lot of reasons why a shutdown might be improper and result in a failing filesystem; the number one reason is probably just power loss or shutting off power without software shutdown. Errors in the drive itself are another possible reason for filesystem errors (underlying hardware issues instead of software issues).
What you really need to do is post a full serial console boot log. I’m not sure what the hardware is, but most carrier boards will have a port which is in device mode and presents a serial console UART over USB, or alternatively, via the ground, TX, and RX pins on a header. Default setup for serial UART might depend on the carrier board, but the software setting would be 115200 8N1.
One particular point about this filesystem error is that it is init itself which is failing. During boot you could consider the bootloader its own operating system, and its one goal is to run the Linux kernel to overwrite itself. When the Linux kernel starts, it is like many programs whereby it takes arguments and perhaps inherits from the existing hardware state. The kernel then runs one (and only one) program: init. It is init which spawns all those other processes and threads (init is PID 1, the kernel is PID 0). For Ubuntu init is a symbolic link to systemd. Since it is init which cannot run it is impossible for further boot without a panic. It would be important to find out what happened to keep this from happening again, but as is, the way I see it is that you’ll have to reflash.
You can attempt to clone your storage device, and perhaps recover parts of it. You’d have to give much more detail on what the storage device is, e.g., eMMC, NVMe, USB storage, SD card, so on, before any advice could be given for that.
