How to generate Trace output for Drive AGX

ali.sevindik · July 30, 2021, 9:14am

Hi,
I’d like to use wcet (worst case execution time) analyzer on NVIDIA Drive AGX board. Here are my uname output and the platforms I am using:
uname: Linux tegra-ubuntu 4.14.102-rt53-tegra #1 SMP PREEMPT RT Fri Sep 20 16:23:45 PDT 2019 aarch64 aarch64 aarch64 GNU/Linux

NVIDIA DRIVE™ Software 10.0 (Linux)
Target Operating System
Linux

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)

SDK Manager Version
1.5.1.7814
Host Machine Version
native Ubuntu 18.04

According to analyzer Vendor, they specify that for the arm (for example, armv8 aarch64 ) tool needs trace output. The format is CoreSight - Embedded Trace Macrocell (ETM) instruction trace data,
e.g. generated by
** ARMv7/v8 AARch32, e.g., Cortex-A53, Cortex-R5F**
However I couldn’t manage to generate this trace on AGX board. I have seen this issue:Installing Linux performance monitoring tools on Jetson TX2 and followed the same procedure to install perf tool. In the link here :Coresight - HW Assisted Tracing on ARM — The Linux Kernel documentation
It is stated that
There are two ways to use the Coresight framework:

using the perf cmd line tools.
interacting directly with the Coresight devices using the sysFS interface.

So I tried option 1 which is perf cmd lien tools. However, In the linke above from nvidia forum, when I run the
./perf list pmu command the output is : List of pre-defined events (to be used in -e):

which is empty!. But I think it should list as cs_etm// [Kernel PMU event] which corresponds to coresight embedded trace macrocell.

How can I proceed at this moment? Should coresight drivers be installed already? or am I missign something or should I install the drivers through NVIDIA SDK etc.?

Thanks for any help!

VickNV · July 30, 2021, 1:05pm

Hi @ali.sevindik ,

I’m not sure if it has something to do with running on guest os.
What’s your “perf list” output? Does it include any PMU events?

ali.sevindik · July 30, 2021, 1:07pm

Hi,
perf list pmu does not show anything but perf list result is :
`nvidia@tegra-ubuntu:~/l4t/kernel/kernel-4.9/tools/perf$ ./perf list

List of pre-defined events (to be used in -e):

rNNN [Raw hardware event descriptor]
cpu/t1=v1[,t2=v2,t3 …]/modifier [Raw hardware event descriptor]
(see ‘man perf-list’ on how to encode it)

mem:[/len][:access] [Hardware breakpoint]`

VickNV · July 30, 2021, 1:10pm

What did you see with “perf list sw”?

ali.sevindik · July 30, 2021, 1:12pm

Hi @VickNV

Note that : I should’ve seen the cs_etm// [Kernel PMU event] result as it is shown in the Coresight - HW Assisted Tracing on ARM — The Linux Kernel documentation link when I typed ./perf list pmu
Anyhow, perf list sw output is:
nvidia@tegra-ubuntu:~/l4t/kernel/kernel-4.9/tools/perf$ sudo ./perf list sw

List of pre-defined events (to be used in -e):

alignment-faults [Software event]
bpf-output [Software event]
context-switches OR cs [Software event]
cpu-clock [Software event]
cpu-migrations OR migrations [Software event]
dummy [Software event]
emulation-faults [Software event]
major-faults [Software event]
minor-faults [Software event]
page-faults OR faults [Software event]
task-clock [Software event]

VickNV · July 30, 2021, 1:43pm

I’m not familiar with CoreSight tools. Maybe its kernel drivers are needed.
Could you check and enable it if needed?

ali.sevindik · July 30, 2021, 3:07pm

@VickNV Could you help how to check/enable etc.?

VickNV · July 30, 2021, 5:13pm

I assume you will need coresight drivers at least.
You can check with below command on the target system.

$ gunzip -c /proc/config.gz | grep -i CORESIGHT

If no, you may need to enable some kernel drivers under ~/nvidia/nvidia_sdk/DRIVE_Software_10.0_Linux_OS_DDPX/DRIVEOS/drive-oss-src/kernel/drivers/hwtracing/coresight .
Better check if any guidance for this in the internet.

ali.sevindik · August 3, 2021, 4:06pm

Hi @VickNV,
output of the gunzip -c /proc/config.gz | grep -i CORESIGHT is : # CONFIG_CORESIGHT is not set
I have the kernel drivers under that path but they are not inside the AGX board. I know that everything is selected via SDK manager. How should I enable coresight drivers on Linux(target) side?

VickNV · August 3, 2021, 8:44pm

Please refer to Compiling the Kernel (NVIDIA DRIVE Linux) (source is under ~/nvidia/nvidia_sdk/DRIVE_Software_10.0_Linux_OS_DDPX/DRIVEOS/drive-oss-src so just start from “To compile the kernel”). Thanks.

ali.sevindik · August 5, 2021, 12:34pm

Hi @VickNV , Thank you a lot for the information.
Problem solved by adding this: source "drivers/hwtracing/coresight/Kconfig" to Kconfig file under ~/nvidia/nvidia_sdk/DRIVE_Software_10.0_Linux_OS_DDPX/DRIVEOS/drive-oss-src/kernel/drivers/Kconfig. Before the step 5, run make menuconfig under ~/nvidia/nvidia_sdk/DRIVE_Software_10.0_Linux_OS_DDPX/DRIVEOS/drive-oss-src/out-t186ref-linux Activate CoreSight Drivers from the Device Drivers menu, then save and continue the steps.
Thank you for the help.

VickNV · August 5, 2021, 1:23pm

Good to hear you made it.

ali.sevindik · August 6, 2021, 2:02pm

Hi @VickNV again ,

coresight devices are not being shown up under the /sys/bus/coresight/devices. Do you have any idea what is the problem? ./perf list pmu shows cs_etm// but cannot observer devices. Is it something coresight devices registration problem?

VickNV · August 6, 2021, 3:37pm

I found this jetson document states Xavier isn’t supported and doubt the possibility.
But you can take a look at https://www.kernel.org/doc/Documentation/devicetree/bindings/arm/coresight.txt and search the internet to see if other developers have such issues.

ali.sevindik · August 9, 2021, 9:16am

Hi @VickNV ,

I think similar issue is observed on NXP side as well: Solved: Re: Enabling CoreSight on i.MX 8MM - NXP Community
I think We need a .dtsi file for coresight on armv8 for NVIDIA DRIVE AGX board. Could you please provide us the similar .dtsi and the process we should follow to enable devices?
Thanks a lot.

VickNV · August 10, 2021, 7:24pm

We will check internally and get back to you. Thanks.

VickNV · August 16, 2021, 4:05pm

“Carmel” CPU doesn’t support ETM/PTM.
That’s why Xavier is excluded in the document I pointed out in How to generate Trace output for Drive AGX - #17 by VickNV.

ali.sevindik · August 18, 2021, 2:59pm

Hi @VickNV,
Thank you fvery much for your time. We have also Jetson board as well. Is it possible to share similar guideline for kernel build + device tree configuration on Jetson ? We can try the same things on Jetson.
thanks a lot.

VickNV · August 18, 2021, 3:19pm

It’s not supported for Carmel CPUs which is the CPU complex in Xaiver SoC (used in both Jetson and DRIVE AGX devkits).
That’s why the Jetson document, I pointed out in How to generate Trace output for Drive AGX - #17 by VickNV, excludes Jetson Xavier.