TX1 Watchdog issue whilst debugging with Lauterbach?

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Hi

We are developing a video codec on an FPGA board attached to the PCIe slot on the Jetson TX1 board. The idea is to send video YUV422_IYUYV frames over the PCIe interface and receive coded video frames back.

We have had issues with our FPGA card preventing the TX1 from booting so we have had to disable msi interrupts in order for the TX1 to boot. Once it is booted we install our FPGA driver and I can successfully send video over to the FPGA.

Because MSI is disabled at boot the FPGA has to generate a legacy PCI interrupt (INTA) to signal a transfer being complete. Unfortunately this causes an immediate and catastrophic crash of my TX1. Both the serial console and the Ubuntu GUI hang immediately.

I’ve previously used the Lauterbach debugger to analyse these sort of issues so we purchased the TEGRAX1 license in the hope that we could catch the problem in the debugger.

So I’ve set up the Lauterback to attach to the running kernel and I can debug no problem but if I stop the TX1 and start single stepping after about two minutes the TX1 reboots. My assumption is that some sort of watchdog is kicking in but I don’t know from where or how to disable it.

Any ideas?

2

Ok, I figured this out

echo 1 > /sys/kernel/debug/tegra_wdt/disable_wdt_reset

3

I would be very interested in what scripts you used to attach the Lauterbach, or any information on the requirements for setup (I’m trying to set up a Trace32 under a Linux host).

4

I’m using windows but the same stuff applies to Linux :-)

Number of things really.

  1. Downloaded the NVIDIA Tegra Linux Driver Package Development Guide and unpack

    [url]https://developer.nvidia.com/embedded/dlc/l4t-documentation-24-2[/url]

  2. Unpack the above link and open the Start_L4T_Docs.html file in a browser

  3. Click “Lauterbach Debuging Scripts” on the menu on the left then click the “Lauterbach scripts” link just below the heading. This will get you lauterbach.tar.gz.

  4. Unpack this and extract the files into a convenient directory for use later

  5. For the TX1 you must create a root user to enable the debugging in sysfs (sudo does not work!)

  • On the Jetson TX1

$ sudo passwd root

Enter and confirm a root password (you only need to do this once obviously)

Start a root shell

$ su

$ echo 0 > /sys/devices/system/cpu/cpuquiet/tegra_cpuquiet/enable
$ echo 1 > /sys/kernel/debug/cpuidle_t210/fast_cluster_states_enable
$ echo 1 > /sys/kernel/debug/cpuidle_t210/slow_cluster_states_enable
$ echo 1 > /sys/kernel/debug/clock/csite/state
$ echo 1 > /sys/kernel/debug/tegra_wdt/disable_wdt_reset

$ exit

You will need to do this part every time you reset/power cycle the board. I haven’t found a way to automatically open a root shell and run this script. You could recompile the kernel/dtb to setup like this I suppose?

  1. You will need to make a clean install of the Trace32 software from Lauterbach and install only the arm64 components. Apparently if you mix arm and arm64 things go horribly wrong (according to Lauterbach uk)

  2. Attach the Lauterbach to the board and start Trace32

  3. Nvidia supply a script in the tar.gz called cpu_kernel_attach.cmm. You need to modify this script to suit where you have installed the Trace32 software and where your kernel source and vmlinux is. I build the Nvidia stuff on a Linux Mint VM as I hate Ubuntu UI and I have the home directory as a samba share in windows Below is my version of the script…

;-------------------------------------------------------------------------------
; Copyright (c) 2012-2014, NVIDIA CORPORATION. All rights reserved.
;-------------------------------------------------------------------------------
; Lauterbach TRACE32 debugger script to set user-specific parameters:
;
; For NVIDIA internal use only. Customers should not use this script
; unless NVIDIA explicitly directs them to do so.
;
; Lauterbach TRACE32 debugger script to attach to CPU with kernel symbols loaded.

; Path to compiled kernel
&KERNEL_ELF=“Z:\Sandbox\Islay\64_TX1\Linux_for_Tegra_64_tx1\sources\kernel_source\vmlinux”

; Path to the lauterbach scripts
&TARGET_SOC_PATH=“C:\Sandbox_git_new\LauterbachScripts_14_6\tx1”

; TX1 model
&TARGET_SOC=“t21x”

; Directory you installed Trace32 to
&T32SYS=“C:\t32”

; Setup JTAG for multicore CPU.
;do &(TARGET_SOC_PATH)/&(TARGET_SOC)/&(TARGET_SOC)_cpu_mp_jtag_setup.cmm

; Setup JTAG for single core CPU.
do &(TARGET_SOC_PATH)/&(TARGET_SOC)/&(TARGET_SOC)_cpu_jtag_setup.cmm

global &LINUX_AWARENESS

; Setup for kernel virtual addressing mode.
if “&TARGET_SOC”!=“t21x”
(
do virtual_setup.cmm
&LINUX_AWARENESS=“&T32SYS/demo/arm/kernel/linux/linux-3.x”
)
else
(
&LINUX_AWARENESS=“&T32SYS/demo/arm64/kernel/linux/linux-3.x”
)

sYmbol.RESet

; Load the kernel symbols symbols. The strippart variable removes components from the path until
; you get the path to be the relative to your kernel source tree
; On my system /strippart 9 removes the "Z:\Sandbox\Islay\64_TX1\Linux_for_Tegra_64_tx1\sources\kernel_source" from the path
print “Loading &KERNEL_ELF symbols”
data.load.elf &KERNEL_ELF /NOCODE /gnu /strippart 9 /lowerpath

IF “&BUILDTOP”!=“”
(
SYMBOL.SOURCEPATH.TRANSLATE “&BUILDTOP” “&TOP”
)

IF (“&LOAD_SAVED_WINDOWS”==“TRUE”)
(
; Restore saved window configuration (this is always saved in the $T32SYS directory – don’t change the path)
do &T32SYS/windows.cmm
)

system.mode.attach

task.setdir &LINUX_AWARENESS
task.config &LINUX_AWARENESS/linux3
menu.ReProgram &LINUX_AWARENESS/linux.men
if “&TARGET_SOC”==“t21x”
(
menu.ReProgram &T32SYS/menarmv8a.men
)

; Optionally load saved breakpoints
;do &TARGET_SOC_PATH/…/breaks.cmm

go