Need help in getting KGDB to work

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hello,

I am trying to get the KGDB going for AGX Xavier. Currently, my serial connection(minicom) to the Xavier box is through the micro-USB FTDI serial device (/dev/ttyTCU0).

root@magneto:~# cat /proc/cmdline 
root=/dev/mmcblk0p1 rw rootwait console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 OS=l4t memtype=0 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x140000@0x96183000 lut_mem=0x2008@0x9607e000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 tegra_keep_boot_clocks maxcpus=8 boot.slot_suffix= boot.ratchetvalues=0.2.2 sdhci_tegra.en_boot_part_access=1  
root@magneto:~#

But when I get to the KDB prompt by following below steps:

echo ttyTCU0 > /sys/module/kgdboc/parameters/kgdboc
echo g > /proc/sysrq-trigger

the system doesn’t accept any input from the keyboard. So, I have to hard reset the system. I think that the driver for ttyTCU0 needs similar hooks as ttyS0 has. Can you please guide me in fixing this?

Also, I tried connecting the Adafruit USB to TTL Serial Cable – Debug / Console Cable for Raspberry Pi cable from a TX2 to the 40-pin expansion header. But I don’t see anything on the minicom opened for that /dev/ttyUSB* port. I have set console=ttyS0, 115200n8 in the kernel boot parameter through flash.sh script. Am I missing something here? I can provide photo of the connection I have made, but it is similar to that of TX2’s with Pin1 assumed to be the upper-right pin of the expansion header.

Please let me know if more information is required.

2

I haven’t tried kgdb yet on the Xavier, but what is the output from:

zcat /proc/config.gz | grep KGDB
3

Hi linuxdev,

Here’s more info on this issue:

root@magneto:~# zcat /proc/config.gz | egrep 'KGDB|CONSOLE_POLL'
# CONFIG_SERIAL_KGDB_NMI is not set
CONFIG_CONSOLE_POLL=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
CONFIG_KGDB_KDB=y
root@magneto:~#

The dmesg output showing the driver used:

[    7.571901] console [ttyTCU0] enabled

{kernel_src}/nvidia/drivers/tty/serial/tegra-combined-uart.c:333

static struct uart_ops tegra_combined_uart_ops = {
	.pm		= (void (*)(struct uart_port *,
					unsigned int,
					unsigned int)) &uart_null_func,
	.tx_empty	= tegra_combined_uart_tx_empty,
	.get_mctrl	=
		(unsigned int (*)(struct uart_port *)) &uart_null_func,
	.set_mctrl	= (void (*)(struct uart_port *,
					unsigned int)) &uart_null_func,
	.stop_tx	= (void (*)(struct uart_port *)) &uart_null_func,
	.start_tx	= tegra_combined_uart_start_tx,
	.stop_rx	= (void (*)(struct uart_port *)) &uart_null_func,
	.break_ctl	= (void (*)(struct uart_port *, int)) &uart_null_func,
	.startup	= tegra_combined_uart_startup,
	.shutdown	= (void (*)(struct uart_port *)) &uart_shutdown,
	.set_termios	= (void (*)(struct uart_port *,
					struct ktermios *,
					struct ktermios *)) &uart_null_func,
	.type		=
		(const char * (*)(struct uart_port *)) &uart_null_func,
	.release_port	= (void (*)(struct uart_port *)) &uart_null_func,
	.request_port	= uart_null_func,
	.config_port	= (void (*)(struct uart_port *, int)) &uart_null_func,
	.verify_port	= (int (*)(struct uart_port *,
				struct serial_struct *)) &uart_null_func,

#ifdef CONFIG_CONSOLE_POLL
	.poll_get_char = uart_null_func,
	.poll_put_char = (void (*)(struct uart_port *,
					unsigned char)) &uart_null_func,
#endif
};

So, almost all functions are null (return 0) in terms of physical hardware functionalities. If I am not mistaken, the poll_get_char() and poll_put_char() functions are used during KGDB.

This functionality works on Jetson TX2 and TX1 where ttyS0 is used as console.

4

I’m going to admit that I really want to wait until the cloning app is available before I flash or add a custom kernel to my Xavier…I am trying to keep a path to restore any release on all of my embedded systems. Because of the way the device tree and kernel don’t have any kind of “alternate selection at boot” and no clone/restore path exists yet, I’m having trouble convincing myself to experiment and throw in an experimental kernel configured for kgdb/kgdboc. I feel the need to not move to the next release until I can fully reload the existing release (I have backup/restore and everything needed to go forwards/backwards for every L4T release since the TK1, and a few releases from even before that in the L4T 16.x era). As soon as I flash something new and can’t go back I know I’ll regret it.

On the other hand, it might be useful to have a thread collecting some of this information. See these:
[url]https://kgdb.wiki.kernel.org/index.php/KDB_FAQ[/url]
[url]https://mirrors.edge.kernel.org/pub/linux/kernel/people/jwessel/kdb/kgdbKernelArgs.html[/url]
[url]https://mirrors.edge.kernel.org/pub/linux/kernel/people/jwessel/kdb/usingKDB.html[/url]
[url]https://mirrors.edge.kernel.org/pub/linux/kernel/people/jwessel/kdb/EnableKGDB.html[/url]
[url]https://elinux.org/Kgdb[/url]
[url]https://blackfin.uclinux.org/doku.php?id=linux-kernel:debug:kgdb[/url]

I don’t know if the echo to “/sys” is 100% equivalent to adding “kgdboc=ttyTCU0,115200n8” via the device tree and making this available right from the early boot stage and on. You might try appending this to the command line through the device tree “chosen” entry.

Has anyone from NVIDIA configured yet for kgdboc on the Xavier?

5

Thanks, linuxdev for sharing all these useful links here.

Hope someone from NVIDIA could reply on this.

6

Hi Anish,

To reply on this setup:

Also, I tried connecting the Adafruit USB to TTL Serial Cable – Debug / Console Cable for Raspberry Pi cable from a TX2 to the 40-pin expansion header. But I don’t see anything on the minicom opened for that /dev/ttyUSB* port. I have set console=ttyS0, 115200n8 in the kernel boot parameter through flash.sh script. Am I missing something here? I can provide photo of the connection I have made, but it is similar to that of TX2’s with Pin1 assumed to be the upper-right pin of the expansion header.

[Reply]
We are able to use UART Serial Console using USB to TTL Serial Cable on Xavier.
Please follow these steps.

  1. Start getty with corresponding THS port on target.
    40-pin UART is on UARTA (0x03100000)
    Grep for corresponding TSH from kernel log (Ideally for galen it is THS0)
    /sbin/getty -a ubuntu -L 115200 ttyTHS0 &

  2. Attach any serial communication program (e.g minicom) to the enumerated ttyUSB instance on host
    After connecting the serial to TTL adapter there should be a new ttyUSB port.
    e.g: minicom -D /dev/ttyUSBn (where n is the port number gets enumerated after connecting serial cable on host)

Please let us know if you still have any problem setting up UART.

Thanks & Regards,
Sandipan

7

Hi spatra,

I also realized this in last couple of days that the expansion header has /dev/ttyTHS0. I just made sure that I am able to get serial console running on this device after running your command. Thank you for that.

I am yet to check if KGDB would work on ttyTHS0. Shall I run “echo ttytHS0,115200n8 > /sys/module/kgdboc/parameters/kgdboc” and then “echo g > /proc/sysrq-trigger” in order to get KDB prompt? Does the driver running ttyTHS0 has the functions implemented to get/put characters in debug context?

8

Hi spatra,

I built a debug kernel with KGDB options turned on. Connected the USB to TTL Serial Cable on the expansion header of the Xavier. Opened a minicom on the corresponding /dev/ttyUSBn.

But when I echo’d ttyTHS0 to the /sys/module/kgdboc/parameters/kgdboc, I get below error:

root@magneto:/home/nvidia# echo ttyTHS0 > /sys/module/kgdboc/parameters/kgdboc 
bash: echo: write error: No such device
root@magneto:/home/nvidia#

Since this fails I think setting kernel parameter for kgdboc would also fail. Please try it out on your end and let me know. The issue is not about getting any serial console output but that KGDB should work.