How to generate Trace output for Drive AGX

Hi,
I’d like to use wcet (worst case execution time) analyzer on NVIDIA Drive AGX board. Here are my uname output and the platforms I am using:
uname: Linux tegra-ubuntu 4.14.102-rt53-tegra #1 SMP PREEMPT RT Fri Sep 20 16:23:45 PDT 2019 aarch64 aarch64 aarch64 GNU/Linux

NVIDIA DRIVE™ Software 10.0 (Linux)
Target Operating System
Linux

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)

SDK Manager Version
1.5.1.7814
Host Machine Version
native Ubuntu 18.04

According to analyzer Vendor, they specify that for the arm (for example, armv8 aarch64 ) tool needs trace output. The format is CoreSight - Embedded Trace Macrocell (ETM) instruction trace data,
e.g. generated by
** ARMv7/v8 AARch32, e.g., Cortex-A53, Cortex-R5F**
However I couldn’t manage to generate this trace on AGX board. I have seen this issue:Installing Linux performance monitoring tools on Jetson TX2 and followed the same procedure to install perf tool. In the link here :Coresight - HW Assisted Tracing on ARM — The Linux Kernel documentation
It is stated that
There are two ways to use the Coresight framework:

  1. using the perf cmd line tools.
  2. interacting directly with the Coresight devices using the sysFS interface.

So I tried option 1 which is perf cmd lien tools. However, In the linke above from nvidia forum, when I run the
./perf list pmu command the output is : List of pre-defined events (to be used in -e):

which is empty!. But I think it should list as cs_etm// [Kernel PMU event] which corresponds to coresight embedded trace macrocell.

How can I proceed at this moment? Should coresight drivers be installed already? or am I missign something or should I install the drivers through NVIDIA SDK etc.?

Thanks for any help!

Hi @ali.sevindik ,

I’m not sure if it has something to do with running on guest os.
What’s your “perf list” output? Does it include any PMU events?

Hi,
perf list pmu does not show anything but perf list result is :
`nvidia@tegra-ubuntu:~/l4t/kernel/kernel-4.9/tools/perf$ ./perf list

List of pre-defined events (to be used in -e):

rNNN [Raw hardware event descriptor]
cpu/t1=v1[,t2=v2,t3 …]/modifier [Raw hardware event descriptor]
(see ‘man perf-list’ on how to encode it)

mem:[/len][:access] [Hardware breakpoint]`

What did you see with “perf list sw”?

Hi @VickNV

Note that : I should’ve seen the cs_etm// [Kernel PMU event] result as it is shown in the Coresight - HW Assisted Tracing on ARM — The Linux Kernel documentation link when I typed ./perf list pmu
Anyhow, perf list sw output is:
nvidia@tegra-ubuntu:~/l4t/kernel/kernel-4.9/tools/perf$ sudo ./perf list sw

List of pre-defined events (to be used in -e):

alignment-faults [Software event]
bpf-output [Software event]
context-switches OR cs [Software event]
cpu-clock [Software event]
cpu-migrations OR migrations [Software event]
dummy [Software event]
emulation-faults [Software event]
major-faults [Software event]
minor-faults [Software event]
page-faults OR faults [Software event]
task-clock [Software event]

I’m not familiar with CoreSight tools. Maybe its kernel drivers are needed.
Could you check and enable it if needed?

@VickNV Could you help how to check/enable etc.?

I assume you will need coresight drivers at least.
You can check with below command on the target system.

$ gunzip -c /proc/config.gz | grep -i CORESIGHT

If no, you may need to enable some kernel drivers under ~/nvidia/nvidia_sdk/DRIVE_Software_10.0_Linux_OS_DDPX/DRIVEOS/drive-oss-src/kernel/drivers/hwtracing/coresight .
Better check if any guidance for this in the internet.

Hi @VickNV,
output of the gunzip -c /proc/config.gz | grep -i CORESIGHT is : # CONFIG_CORESIGHT is not set
I have the kernel drivers under that path but they are not inside the AGX board. I know that everything is selected via SDK manager. How should I enable coresight drivers on Linux(target) side?

Please refer to Compiling the Kernel (NVIDIA DRIVE Linux) (source is under ~/nvidia/nvidia_sdk/DRIVE_Software_10.0_Linux_OS_DDPX/DRIVEOS/drive-oss-src so just start from “To compile the kernel”). Thanks.

Hi @VickNV , Thank you a lot for the information.
Problem solved by adding this: source "drivers/hwtracing/coresight/Kconfig" to Kconfig file under ~/nvidia/nvidia_sdk/DRIVE_Software_10.0_Linux_OS_DDPX/DRIVEOS/drive-oss-src/kernel/drivers/Kconfig. Before the step 5, run make menuconfig under ~/nvidia/nvidia_sdk/DRIVE_Software_10.0_Linux_OS_DDPX/DRIVEOS/drive-oss-src/out-t186ref-linux Activate CoreSight Drivers from the Device Drivers menu, then save and continue the steps.
Thank you for the help.

Good to hear you made it.

Hi @VickNV again ,

coresight devices are not being shown up under the /sys/bus/coresight/devices. Do you have any idea what is the problem? ./perf list pmu shows cs_etm// but cannot observer devices. Is it something coresight devices registration problem?

I found this jetson document states Xavier isn’t supported and doubt the possibility.
But you can take a look at https://www.kernel.org/doc/Documentation/devicetree/bindings/arm/coresight.txt and search the internet to see if other developers have such issues.

Hi @VickNV ,

I think similar issue is observed on NXP side as well: Solved: Re: Enabling CoreSight on i.MX 8MM - NXP Community
I think We need a .dtsi file for coresight on armv8 for NVIDIA DRIVE AGX board. Could you please provide us the similar .dtsi and the process we should follow to enable devices?
Thanks a lot.

We will check internally and get back to you. Thanks.

2 Likes

“Carmel” CPU doesn’t support ETM/PTM.
That’s why Xavier is excluded in the document I pointed out in How to generate Trace output for Drive AGX - #17 by VickNV.

Hi @VickNV,
Thank you fvery much for your time. We have also Jetson board as well. Is it possible to share similar guideline for kernel build + device tree configuration on Jetson ? We can try the same things on Jetson.
thanks a lot.

It’s not supported for Carmel CPUs which is the CPU complex in Xaiver SoC (used in both Jetson and DRIVE AGX devkits).
That’s why the Jetson document, I pointed out in How to generate Trace output for Drive AGX - #17 by VickNV, excludes Jetson Xavier.

image