Kernel messages stop after roughly 2 seconds

Dear all,

we are trying to get a custom carrier board to work. The flash process and the bootloader seem to work. Also the Linux kernels begins to boot, but after roughly 2 seconds after leaving the bootloader the output on UART3 from the kernel stops. The last printed message from the kernel changes from boot to boot (see bootlogs). But the Jetson Module seems to continue to boot, as the power consumption increases ca. 10 seconds later by roughly 150 mA.

We enabled in the DeviceTree the UART2 and UART3 and set the pinmux to use the corresponding pins for the UARTs.

Bootlogs:
output_2023-05-08_11-18-44.log (57.5 KB)
output_2023-05-08_11-20-28.log (57.9 KB)

Furthermore, is there some minimal DeviceTree example for a custom carrier board as a reference?

Best,
Gerrit

Is there anything else that can check whether the device is booting up instead of checking the serial console on your side?

For example, HDMI monitor.

Unfortunately we only have Ethernet (link is not coming up, but there are errors regarding the PHY in the bootlog), UART2 and UART3, PCIe and a few GPIO. Your current idea is to toggle a GPIO (using a small program in the autostart).

Sorry for the late response, have you managed to get issue resolved or still need the support? Thanks

We still need support on this issue.
We added scripts to the rc.local to toggle a GPIO or output a heartbeat on every found serial port. But neither showed any reactions. It looks like the system is not starting completely.

Thanks,
Gerrit

I am curious, what do you see from:
systemctl status rc-local.service

@linuxdev That information was unfortunately not available, as the system did not exposed a login prompt.

We tried the DeviceTree of the Development Kit, after checking, that no GPIO in could create a short. With that configuration the board boots up. After adding setsid /sbin/getty -L 115200 ttyTHS1 to be executed at boot-time we also got a login prompt on UART2, while the kernel/systemd message are printed via UART3.
While using the following kernel commandline:
console=ttyTCU0,115200 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS1,115200n8 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt rootfs.slot_suffix= tegra_fbmem=0x800000@0xa06ab000 lut_mem=0x2008@0xa06a5000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix= boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1

In the config file for flashing the following is defined:
CMDLINE_ADD="console=ttyS1,115200n8 fbcon=map:0 net.ifnames=0 rootfstype=ext4";

We disabled the UART3 (uartc) in the device tree and changed the CMDLINE of the kernel in the device tree to use ttyTCU0. Furthermore we added a getty command in the rc.local for ttyTHS1. With these changes, our custom device tree shows the same behavior as the on for the development kit.

This leads me to the following questions:

  1. Why is the console parameter ttyS1 not working? We can also not write to ttyS1 after login:
root@localhost:/home/jetson# echo "Test" > /dev/ttyS1
bash: echo: write error: Input/output error

The same approach works for ttyTHS1. There is also no error in the dmesg log.

  1. Is there some minimal device tree example for a Jetson Xavier AGX with Ethernet and UART2 and UART3 for debuging? To ensure, that no other thinks are missing right now.

Thanks,
Gerrit

I should have paid more attention to the part about not reaching login.

I am now curious though about something else. This is an excerpt of the kernel command line:
console=ttyTCU0,115200 ... console=ttyS1,115200n8

I see no other console entry, so you have two serial consoles. In your device tree, what are all of your “serial@...” entries? Note that the ttyTCU0 isn’t something I’m looking at since this is normally just serial console (I would be careful about using ttyTCU0 for anything except serial console; this hardware did not exist in earlier Jetsons, and has only been used for serial consoles since its existence…I think this UART might not be suitable for general use). The other UARTs can be used for almost anything, and the UARTs with name format “ttyS#” and “ttyTHS#” are the same physical UARTs, but using a different driver. I expect a ttyS# to be available even during bootloader stages (if something enables them), but I expect the ttyTHS# to not be available until after the Linux kernel starts (I don’t think there is a driver available for the ttyTHS# in boot software).

What are all of your “serial@...” device tree entries, and are you certain the physical addresses and pins related to this are consistent? I couldn’t say which one goes to what, but this could cause problems if wrong.

Incidentally, the “/etc/rc.local” file is not always activated in newer Ubuntu releases. I was thinking this might not be enabled for your test, but since things did not get far enough along, there is no way to tell if edits there are active or not.

In the device tree we have only the following entry for the working configuration (continuous messages on UART 3):

serial@3110000 {
	status = "okay";
};

If we add the following interface, the messages stops after 2 seconds:

serial@c280000 {
	status = "okay";
};

The configuration of the pinmux is to use the corresponding pins for the UARTS.

Best,
Gerrit

I don’t know the details of addresses for various serial UARTs, but with that someone from NVIDIA should be able to verify if that is the correct and complete UART device tree setup on the AGX dev kit.

It is a custom carrier board. Therefore only a subset of the UARTs are routed out to pins (UART2 and UART3).

Someone from NVIDIA would have to comment, although I suspect the answer will be to see the board bring-up and customization section of the documentation. They can answer questions about specific UART device tree content, but that is something I don’t have knowledge of.

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Sorry for late reply.

Could someone share a brief summary about what gets changed here?