Amelie
August 22, 2023, 2:58am
1
I have read this Debugging the Kernel tutorial. But I’m not sure which tty is used for debugging the kernel.
ran the following command, but encountered an issue when configuring kgdboc.
root@ubuntu:~# dmesg | grep tty
[ 0.000000] Kernel command line: root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=efifb:off nospectre_bhb nv-auto-config
[ 0.001753] printk: console [tty0] enabled
[ 2.360982] printk: console [ttyTCU0] enabled
[ 7.393575] 3100000.serial: ttyTHS0 at MMIO 0x3100000 (irq = 35, base_baud = 0) is a TEGRA_UART
[ 7.408433] 3110000.serial: ttyTHS1 at MMIO 0x3110000 (irq = 36, base_baud = 0) is a TEGRA_UART
[ 7.423188] 3140000.serial: ttyTHS4 at MMIO 0x3140000 (irq = 37, base_baud = 0) is a TEGRA_UART
[ 10.277928] systemd[1]: Created slice system-serial\x2dgetty.slice.
root@ubuntu:~# echo ttyTHS0,115200 > /sys/module/kgdboc/parameters/kgdboc
-bash: echo: write error: No such device
.config :
root@ubuntu:~# zcat /proc/config.gz | egrep 'CONFIG_KGDB|KDGB|CONFIG_FRAME_POINTER|CONFIG_WATCHDOG|IKCONFIG'
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
CONFIG_WATCHDOG_NOWAYOUT=y
CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=y
CONFIG_WATCHDOG_OPEN_TIMEOUT=0
# CONFIG_WATCHDOG_SYSFS is not set
# CONFIG_WATCHDOG_PRETIMEOUT_GOV is not set
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_HONOUR_BLOCKLIST=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
CONFIG_KGDB_KDB=y
Amelie
August 22, 2023, 5:29am
3
When I use ttyS0 as a debugging interface, extlinux.conf is as follows:
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=efifb:off nospectre_bhb nv-auto-config console=ttyS0,115200n8 kgdboc=ttyS0,115200 kgdbwait
The kernel output is as follows:
[ 2.313088] printk: console [ttyTCU0] enabled
[ 2.318641] ------------[ cut here ]------------
[ 2.321823] WARNING: CPU: 3 PID: 1 at drivers/tty/serial/serial_core.c:483 uart_get_baud_rate+0xf0/0x180
[ 2.331085] Modules linked in:
[ 2.334069] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 5.10.120-hello-tegra #2
[ 2.341411] Hardware name: Unknown Jetson-AGX/Jetson-AGX, BIOS 202307.0-e447998 08/17/2023
[ 2.349639] pstate: 20800009 (nzCv daif -PAN +UAO -TCO BTYPE=--)
[ 2.355759] pc : uart_get_baud_rate+0xf0/0x180
[ 2.359878] lr : uart_get_baud_rate+0x8c/0x180
[ 2.364595] sp : ffff80001004b7e0
[ 2.367749] x29: ffff80001004b7e0 x28: ffffda205371dc50
[ 2.373086] x27: ffff80001004b95c x26: 00000000ffffffff
[ 2.378854] x25: 0000000000009600 x24: 0000000000000001
[ 2.384024] x23: 0000000000000001 x22: 0000000000000000
[ 2.389448] x21: 0000000000000000 x20: 0000000000000000
[ 2.394962] x19: 0000000000009600 x18: 0000000000000000
[ 2.400468] x17: 0000000031dfb1f3 x16: 00000000ff207ed2
[ 2.405724] x15: ffffffffffffffff x14: 0000000000000000
[ 2.411405] x13: 0000000000000001 x12: 0000000000000000
[ 2.416830] x11: ffffda2052b1fa78 x10: 0000000000000000
[ 2.422088] x9 : 0000000000000000 x8 : 0000000000000000
[ 2.427600] x7 : 0000000000000000 x6 : 000000000000001e
[ 2.433368] x5 : ffffda2052b1f9f8 x4 : 00000000003d0900
[ 2.438537] x3 : 000000000000001f x2 : 00000000ffffeff0
[ 2.444151] x1 : 0000000000000cb0 x0 : 0000000000002580
[ 2.449210] Call trace:
[ 2.451665] uart_get_baud_rate+0xf0/0x180
[ 2.455711] serial8250_get_baud_rate+0x98/0xb0
[ 2.460414] serial8250_do_set_termios+0xd8/0x440
[ 2.464961] serial8250_set_termios+0x54/0x70
[ 2.469528] uart_set_options+0x114/0x190
[ 2.473534] uart_poll_init+0x130/0x150
[ 2.477042] tty_find_polling_driver+0x12c/0x1f0
[ 2.481768] configure_kgdboc+0xf8/0x1f0
[ 2.485784] kgdboc_probe+0x54/0x70
[ 2.489286] platform_drv_probe+0x5c/0xb0
[ 2.493309] really_probe+0xf8/0x3d0
[ 2.496753] driver_probe_device+0x60/0xc0
[ 2.501008] __device_attach_driver+0x8c/0xd0
[ 2.505295] bus_for_each_drv+0x8c/0xe0
[ 2.508823] __device_attach+0xf8/0x160
[ 2.512995] device_initial_probe+0x28/0x40
[ 2.516846] bus_probe_device+0xa4/0xb0
[ 2.520528] device_add+0x364/0x750
[ 2.524369] platform_device_add+0x108/0x240
[ 2.528662] init_kgdboc+0x5c/0x90
[ 2.531839] do_one_initcall+0x50/0x2f0
[ 2.535932] kernel_init_freeable+0x254/0x2d4
[ 2.539957] kernel_init+0x1c/0x128
[ 2.543543] ret_from_fork+0x10/0x18
[ 2.547140] ---[ end trace 6ae1fc05078f116a ]---
[ 2.552003] Unable to handle kernel paging request at virtual address fffffdfffec00001
[ 2.559634] Mem abort info:
[ 2.562435] ESR = 0x96000046
[ 2.565589] EC = 0x25: DABT (current EL), IL = 32 bits
[ 2.570923] SET = 0, FnV = 0
[ 2.574072] EA = 0, S1PTW = 0
[ 2.577132] Data abort info:
[ 2.580193] ISV = 0, ISS = 0x00000046
[ 2.584130] CM = 0, WnR = 1
[ 2.586936] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000bdf29000
[ 2.593841] [fffffdfffec00001] pgd=00000000be652003, p4d=00000000be652003, pud=00000000be653003, pmd=0000000000000000
[ 2.604344] Internal error: Oops: 96000046 [#1] PREEMPT SMP
[ 2.609599] Modules linked in:
[ 2.612663] CPU: 3 PID: 1 Comm: swapper/0 Tainted: G W 5.10.120-hello-tegra #2
[ 2.621493] Hardware name: Unknown Jetson-AGX/Jetson-AGX, BIOS 202307.0-e447998 08/17/2023
[ 2.629376] pstate: 60800089 (nZCv daIf -PAN +UAO -TCO BTYPE=--)
[ 2.635326] pc : io_serial_out+0x4c/0x60
[ 2.639346] lr : io_serial_out+0x28/0x60
[ 2.642935] sp : ffff80001004b840
[ 2.646348] x29: ffff80001004b840 x28: ffffda205371dc50
[ 2.652031] x27: 0000000000000000 x26: 0000000000000000
[ 2.657368] x25: 0000000000000008 x24: 0000000000000000
[ 2.662799] x23: 0000000000000000 x22: ffff10eac013e740
[ 2.668047] x21: 0000000000000000 x20: ffffda205390a588
[ 2.673732] x19: fffffdfffec00001 x18: 0000000000000000
[ 2.678725] x17: 0000000031dfb1f3 x16: 00000000ff207ed2
[ 2.684494] x15: ffffffffffffffff x14: 0000000000000000
[ 2.689749] x13: 0000000000000001 x12: 0000000000000000
[ 2.695261] x11: ffffda2052b1fa78 x10: 0000000000000000
[ 2.700943] x9 : 0000000000000000 x8 : 0000000000000000
[ 2.706456] x7 : 0000000000000000 x6 : 000000000000001e
[ 2.711967] x5 : ffffda2052b1f9f8 x4 : 00000000003d0900
[ 2.717136] x3 : ffffda2052101270 x2 : ffffda20519a2850
[ 2.722730] x1 : 0000000000000000 x0 : fffffdfffec00000
[ 2.727808] Call trace:
[ 2.730264] io_serial_out+0x4c/0x60
[ 2.733938] serial8250_do_set_termios+0x244/0x440
[ 2.739006] serial8250_set_termios+0x54/0x70
[ 2.743036] uart_set_options+0x114/0x190
[ 2.747062] uart_poll_init+0x130/0x150
[ 2.751081] tty_find_polling_driver+0x12c/0x1f0
[ 2.755375] configure_kgdboc+0xf8/0x1f0
[ 2.759572] kgdboc_probe+0x54/0x70
[ 2.763071] platform_drv_probe+0x5c/0xb0
[ 2.767096] really_probe+0xf8/0x3d0
[ 2.770769] driver_probe_device+0x60/0xc0
[ 2.774820] __device_attach_driver+0x8c/0xd0
[ 2.778855] bus_for_each_drv+0x8c/0xe0
[ 2.782845] __device_attach+0xf8/0x160
[ 2.786550] device_initial_probe+0x28/0x40
[ 2.790552] bus_probe_device+0xa4/0xb0
[ 2.794573] device_add+0x364/0x750
[ 2.798070] platform_device_add+0x108/0x240
[ 2.802196] init_kgdboc+0x5c/0x90
[ 2.805684] do_one_initcall+0x50/0x2f0
[ 2.809278] kernel_init_freeable+0x254/0x2d4
[ 2.813823] kernel_init+0x1c/0x128
[ 2.817408] ret_from_fork+0x10/0x18
[ 2.821004] Code: d2bfd800 f2dfbfe0 f2ffffe0 8b000273 (39000275)
[ 2.827305] ---[ end trace 6ae1fc05078f116b ]---
[ 2.831844] Kernel panic - not syncing: Oops: Fatal exception
[ 2.837103] SMP: stopping secondary CPUs
[ 2.841230] Kernel Offset: 0x5a20417d0000 from 0xffff800010000000
[ 2.847416] PHYS_OFFSET: 0xffffef1640000000
[ 2.851275] CPU features: 0x18240002,03802a30
[ 2.855647] Memory Limit: none
[ 2.858808] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
Hi Amelie,
Are you using the devkit or custom board for AGX Xavier?
Do you mean the serial console to debug kernel?/dev/ttyTCU0, which means Tegra Combined UART.
Amelie
August 22, 2023, 7:12am
5
I am using an original embedded development board.
I am using JetPack 5.1.2, and I haven’t used a devkit yet. I am just interested in learning how to debug the kernel on arm64.
Jetson Linux r35.4.1
kernel: kernel-5.10.120-tegra
And I changed exlinux to:
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200n8 console=tty0 kgdboc=ttyTCU0,115200n8 kgdbwait fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=efifb:off nospectre_bhb nv-auto-config
Excerpt:
console=ttyTCU0,115200n8 console=tty0 kgdboc=ttyTCU0,115200 kgdbwait
But I didn’t get any terminal output like:
KGDB: Waiting for connection from remote gdb...
bootloader log: bootloader.txt (66.3 KB)
Amelie
August 22, 2023, 8:11am
6
There is a question here, why is ttyTCU0 set to a baud rate of 9600?
root@ubuntu:~# stty -F /dev/ttyTCU0
speed 9600 baud; line = 0;
-brkint ixoff -imaxbel iutf8
-iexten
In extlinux, it is written as:
console=ttyTCU0,115200n8
Amelie
August 22, 2023, 9:24am
7
Can you provide me with the correct kernel parameter syntax, thanks?
I have not used KGD/KGDBOC in a long time, but it does go through the same serial port that has the serial console. If the normal serial console uses 115200 8N1 (and it does normally), then this should be used. Some of the content it sends is just plain text, e.g., to “/proc/sysrq-trigger”, so it has to perform an ordinary command line echo to that (or other) files.
Amelie
August 23, 2023, 12:35am
9
So I modified extlinux as follows:
LABEL amelie_debug
MENU LABEL amelie debug kernel
LINUX /boot/Image-5.10.120-amelie-tegra
FDT /boot/dtb/kernel_tegra194-p2888-0001-p2822-0000.dtb
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=efifb:off nospectre_bhb nv-auto-config console=ttyTCU0,115200n8 kgdboc=ttyTCU0,115200 kgdbwait
But KGDB still doesn’t work.
I am using minicom to connect to /dev/ttyUSB3, but the minicom terminal does not display the following:
KGDB: Waiting for connection from remote gdb...
I suspect the problem lies here (terminal command error) :
root@ubuntu:~# echo ttyTCU0,115200 > /sys/module/kgdboc/parameters/kgdboc
-bash: echo: write error: No such device
Which port on the AGX board have you connected to your PC ?
Amelie
August 23, 2023, 12:58pm
12
@b-sathishkumar
In AGX, ttyTCU0 is used, while in PC, ttyUSB3 is used.
I really wish I could remember more of this, it has been many years since I’ve used it (the Xavier did not even exist back then). One thing I do remember though is that it is different to catch kgdb/kgdboc upon kernel start compared to when a kernel is already running. There are commands you can echo to the sysrq-trigger once the system has started, versus the command line for telling the kernel to go into the debugger right from the start.
The actual documentation I’ve used before is from kernel.org :https://mirrors.edge.kernel.org/pub/linux/kernel/people/jwessel/kdb/kgdbKernelArgs.html
If you cannot boot directly into kgdb, then you might try triggering it from the running kernel. If you are debugging just one driver, then this is probably the better method anyway. If you are debugging the IRQ vector table (or triggering on it), then you would need to do this at kernel boot.
Here’s another reference:https://elinux.org/Kgdb
1 Like
Since you are using ttyTCU0 for both console as well as kgdboc, can you try using agent-proxy ? (refer below link)
https://www.marcusfolkesson.se/blog/debug-kernel-with-kgdb/
1 Like
system
September 13, 2023, 3:14am
16
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.