Arm CoreSight ETM Tracing not working

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

Hi,

I have a similar problem like Coresight STM and ETF not working on Orin AGX.

For CoreSight Tracing using the Embedded Trace Macrocells (ETMs) I followed the steps described here (please read).
I configured the CoreSight kernel drivers and the Devicetree so that all the devices are shown at /sys/bus/coresight/devices/. Turning on the ETF and ETM via echo 1 > enable_sink/source also works as expected
However, when trying to dump the trace data from the ETF into a file, there is nothing to be written:

dd if=/dev/72030000.etf of=~/trace.bin
0+1 records in
0+1 records out
64 bytes copied, 0.000981324 s, 65.2 kB/s

Using the mem_parser only shows the following:

I give up, asking kernel team for sku id variable name.
mem_parser version 2.0.9

Parsing Pogram Trace Macrocell buffer (ETM4.x) ......
Please note: Direct branch could have been executed before the flush and not after the requested flush.

etb buffer size: 64

Help is very much needed!

Kind regards,
Vincent

3

Hi 1408,

Are you using the devkit or custom board for Jetson Nano?
What’s the Jetpack version in use?

Please share the detailed steps with full commands you used and logs for further check.

4

Hi,

I am using the devkit (4GB RAM, revision A02) with L4T 32.7.5, so JetPack 4.6.5.

For activating the kernel drivers, I followed the steps here (https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3275/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#) with using make nconfig to add the CoreSight drivers.
For modifying the devicetree, I used the DT bindings here (coresight.txt « arm « bindings « devicetree « Documentation - kernel/git/torvalds/linux.git - Linux kernel source tree) with the addresses shown in the Tegra X1 SoC TRM.

The commands for ETM tracing like shown in the documentation:

# echo 1 > /sys/bus/coresight/devices/72030000.etf/72030000.etf/enable_sink
# echo 1 > /sys/bus/coresight/devices/73440000.etm0/73440000.etm0/enable_source
# dmesg
[ 3171.085440] coresight-tmc 72030000.etf: TMC-ETB/ETF enabled
[ 3171.085455] coresight-funnel 72010000.funnel_major: FUNNEL inport 2 enabled
[ 3171.085468] coresight-funnel 73010000.funnel_ccplex: FUNNEL inport 1 enabled
[ 3171.085662] coresight-etm4x 73440000.etm0: ETM tracing enabled
# dd if=/dev/72030000.etf of=~/trace.bin
0+1 records in
0+1 records out
64 bytes copied, 0.000981324 s, 65.2 kB/s
# dmesg
[ 3716.865028] coresight-tmc 72030000.etf: TMC read start
[ 3716.867437] coresight-tmc 72030000.etf: TMC read end
# ~/./mem_parser_arm --formatter --etb ~/trace.bin --elf ~/Downloads/kernel_out/vmlinux
I give up, asking kernel team for sku id variable name.
mem_parser version 2.0.9

Parsing Pogram Trace Macrocell buffer (ETM4.x) ......
Please note: Direct branch could have been executed before the flush and not after the requested flush.

etb buffer size: 64

It seems to me that there is no data written into /dev/72030000.etf.

5

Hi @KevinFFF,

do you have any updates? I have no idea why it doesn’t work.

6

Isn’t there anyone at Nvidia who can help me?

7

Sorry for the late reply that I didn’t get a chance to reproduce it locally.
I will do that by next week.

Could you also verify with the latest Jetpack 4.6.6(r32.7.6) on the Jetson Nano devkit?
Please also share the result of the following command on your board.

$ cat /etc/nv_tegra_release
$ cat /etc/nv_boot_control.conf
9

Turns out I actually have the latest revision running:

$ cat /etc/nv_tegra_release 
# R32 (release), REVISION: 7.6, GCID: 38171779, BOARD: t210ref, EABI: aarch64, DATE: Tue Nov  5 07:46:14 UTC 2024

$ cat /etc/nv_boot_control.conf 
TNSPEC 3448-200-0000-F.0-1-0-jetson-nano-qspi-sd-mmcblk0p1
COMPATIBLE_SPEC 3448-200-0000--1--jetson-nano-devkit-
TEGRA_CHIPID 0x21
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

If you want to reproduce it locally, you would have to use my CoreSight devicetree configurations. I can send you the .dtsi files if you want. Or does Nvidia have their own Coresight devicetree file which was not released?

10

Have you checked if the ETM is actually generating trace data?

How would I check that other than trying to read the data which was written (or in this case: not written) into the ETF?

Maybe try different trace modes or check if there’s a missing configuration in the device tree. Also, could be worth verifying permissions and kernel logs (dmesg) for any errors.

I ran all instructions as root (see the #), and there are no errors in dmesg (otherwise the “enabled” and “read start/end” outputs wouldn’t be there). I also checked the devicetree configuration with a CoreSight expert at Linaro, so this should not be the issue.
One possibility why it doesn’t work could be that the CoreSight drivers are too old (kernel version 4.9) and thus no trace is generated by the ETM or stored in the ETF. But this would not explain why the initialization of the Coresight components is sucsessful.

11

@KevinFFF…

(I also found a driver developed by Nvidia that collects trace data successfully.
However, I have a question about that here)

12

@KevinFFF come on man…
It’s been yet another month

13

Sorry for the late reply.

Okay, please share the full device tree and dmesg for further check.

Have you also checked the following document?

  1. kernel/kernel-4.9/Documentation/trace/coresight.txt
  2. kernel/kernel-4.9/Documentation/devicetree/bindings/arm/coresight.txt
14

Okay, please share the full device tree and dmesg for further check.

I included tegra210-coresight.dtsi in tegra210-soc-base.dtsi and it got compiled into tegra210-p3448-0000-p3449-0000-a02.dtb. You can run

dtc -I dtb -O dts -o devicetree-decompiled.txt tegra210-p3448-0000-p3449-0000-a02.dtb 2> warnings.txt

to decompile the .dtb and store the whole devicetree in devicetree-decompiled.txt.

dmesg.txt (56.5 KB)
tegra210-p3448-0000-p3449-0000-a02.dtb.txt (228.2 KB)
tegra210-coresight.dtsi.txt (7.1 KB)

Have you also checked the following document?

Are you serious?!
The first one is referenced in the guide I linked in my original post (I wrote “please read” for a reason!), and you just confirmed that you did in fact not read it.
The second one is linked in my second post to this thread.

This thread and the others that I created show that the moderators in this forum straight up ignore most of the explanations and links that the original posters provide.
You guys just keep embarrassing yourselves…

15

Sorry that I was not clear about this topic and there’re too many topics on the forum so that I mis-understand what you’ve done.

I’m trying to reproduce your use case locally.
When I perform this to include your device tree, I would get the following errors:

ERROR (phandle_references): Reference to non-existent node or label "cpu0"

ERROR (phandle_references): Reference to non-existent node or label "cpu1"

ERROR (phandle_references): Reference to non-existent node or label "cpu2"

ERROR (phandle_references): Reference to non-existent node or label "cpu3"

Do you hit similar issue before?

16

Okay, I could fix above issue after adding tag for cpu0 to cpu3.

Currently, I can reproduce the exact same result as yours on Jetson Nano devkit.
Please let me check them with internal.

May I know how did you get this coresight configuration for Jetson Nano?

17

I wrote it myself, see the explanation in my 2nd post to this thread:

For modifying the devicetree, I used the DT bindings here (coresight.txt « arm « bindings « devicetree « Documentation - kernel/git/torvalds/linux.git - Linux kernel source tree) with the addresses shown in the Tegra X1 SoC TRM.

18

Still no updates?

19

I’ve digged into this feature with no luck that it is from old release and seems not working currently.
Similar result as tegra_ptm_t210 driver cannot be enabled in current release.

Could you try using ftrace as the following instead?
Please confirm the following kernel configs have been enabled:

CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE

Run the following command to trace:

$ sudo su
# cd /sys/kernel/debug/tracing
# echo 0 > tracing_on
# echo nop > current_tracer
# echo 10240 > buffer_size_kb
# echo > trace
# echo 1 > tracing_on
# ... (run application)
# echo 0 > tracing_on
# cp trace ~/trace-buffer.txt
20

I’ve digged into this feature with no luck that it is from old release and seems not working currently.
Similar result as tegra_ptm_t210 driver cannot be enabled in current release.

As explained in another post, I managed to enable the tegra_ptm_t210 driver with a few small modifications. I just want to know why the standard Linux CoreSight driver does not work. Can you give me a reason?

Could you try using ftrace as the following instead?

ftrace is no option for me, as it does not provide the functionality that I need (instruction-level tracing). Only CoreSight does

22

Do you have a reason why the standard Linux CoreSight driver doesn’t work or not?