Arm CoreSight ETM Tracing not working

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

Hi,

I have a similar problem like Coresight STM and ETF not working on Orin AGX.

For CoreSight Tracing using the Embedded Trace Macrocells (ETMs) I followed the steps described here (please read).
I configured the CoreSight kernel drivers and the Devicetree so that all the devices are shown at /sys/bus/coresight/devices/. Turning on the ETF and ETM via echo 1 > enable_sink/source also works as expected
However, when trying to dump the trace data from the ETF into a file, there is nothing to be written:

dd if=/dev/72030000.etf of=~/trace.bin
0+1 records in
0+1 records out
64 bytes copied, 0.000981324 s, 65.2 kB/s

Using the mem_parser only shows the following:

I give up, asking kernel team for sku id variable name.
mem_parser version 2.0.9

Parsing Pogram Trace Macrocell buffer (ETM4.x) ......
Please note: Direct branch could have been executed before the flush and not after the requested flush.

etb buffer size: 64

Help is very much needed!

Kind regards,
Vincent

3

Hi 1408,

Are you using the devkit or custom board for Jetson Nano?
What’s the Jetpack version in use?

Please share the detailed steps with full commands you used and logs for further check.

4

Hi,

I am using the devkit (4GB RAM, revision A02) with L4T 32.7.5, so JetPack 4.6.5.

For activating the kernel drivers, I followed the steps here (https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3275/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#) with using make nconfig to add the CoreSight drivers.
For modifying the devicetree, I used the DT bindings here (coresight.txt « arm « bindings « devicetree « Documentation - kernel/git/torvalds/linux.git - Linux kernel source tree) with the addresses shown in the Tegra X1 SoC TRM.

The commands for ETM tracing like shown in the documentation:

# echo 1 > /sys/bus/coresight/devices/72030000.etf/72030000.etf/enable_sink
# echo 1 > /sys/bus/coresight/devices/73440000.etm0/73440000.etm0/enable_source
# dmesg
[ 3171.085440] coresight-tmc 72030000.etf: TMC-ETB/ETF enabled
[ 3171.085455] coresight-funnel 72010000.funnel_major: FUNNEL inport 2 enabled
[ 3171.085468] coresight-funnel 73010000.funnel_ccplex: FUNNEL inport 1 enabled
[ 3171.085662] coresight-etm4x 73440000.etm0: ETM tracing enabled
# dd if=/dev/72030000.etf of=~/trace.bin
0+1 records in
0+1 records out
64 bytes copied, 0.000981324 s, 65.2 kB/s
# dmesg
[ 3716.865028] coresight-tmc 72030000.etf: TMC read start
[ 3716.867437] coresight-tmc 72030000.etf: TMC read end
# ~/./mem_parser_arm --formatter --etb ~/trace.bin --elf ~/Downloads/kernel_out/vmlinux
I give up, asking kernel team for sku id variable name.
mem_parser version 2.0.9

Parsing Pogram Trace Macrocell buffer (ETM4.x) ......
Please note: Direct branch could have been executed before the flush and not after the requested flush.

etb buffer size: 64

It seems to me that there is no data written into /dev/72030000.etf.

5

Hi @KevinFFF,

do you have any updates? I have no idea why it doesn’t work.

6

Isn’t there anyone at Nvidia who can help me?

7

Sorry for the late reply that I didn’t get a chance to reproduce it locally.
I will do that by next week.

Could you also verify with the latest Jetpack 4.6.6(r32.7.6) on the Jetson Nano devkit?
Please also share the result of the following command on your board.

$ cat /etc/nv_tegra_release
$ cat /etc/nv_boot_control.conf
8

Sounds like the trace data isn’t being properly captured or flushed. Have you checked if the ETM is actually generating trace data? Maybe try different trace modes or check if there’s a missing configuration in the device tree. Also, could be worth verifying permissions and kernel logs (dmesg) for any errors.

9

Turns out I actually have the latest revision running:

$ cat /etc/nv_tegra_release 
# R32 (release), REVISION: 7.6, GCID: 38171779, BOARD: t210ref, EABI: aarch64, DATE: Tue Nov  5 07:46:14 UTC 2024

$ cat /etc/nv_boot_control.conf 
TNSPEC 3448-200-0000-F.0-1-0-jetson-nano-qspi-sd-mmcblk0p1
COMPATIBLE_SPEC 3448-200-0000--1--jetson-nano-devkit-
TEGRA_CHIPID 0x21
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

If you want to reproduce it locally, you would have to use my CoreSight devicetree configurations. I can send you the .dtsi files if you want. Or does Nvidia have their own Coresight devicetree file which was not released?

10

Have you checked if the ETM is actually generating trace data?

How would I check that other than trying to read the data which was written (or in this case: not written) into the ETF?

Maybe try different trace modes or check if there’s a missing configuration in the device tree. Also, could be worth verifying permissions and kernel logs (dmesg) for any errors.

I ran all instructions as root (see the #), and there are no errors in dmesg (otherwise the “enabled” and “read start/end” outputs wouldn’t be there). I also checked the devicetree configuration with a CoreSight expert at Linaro, so this should not be the issue.
One possibility why it doesn’t work could be that the CoreSight drivers are too old (kernel version 4.9) and thus no trace is generated by the ETM or stored in the ETF. But this would not explain why the initialization of the Coresight components is sucsessful.

11

@KevinFFF…

(I also found a driver developed by Nvidia that collects trace data successfully.
However, I have a question about that here)