Recommended kernel tracing tool

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
#1

I am using L4t kernel with preempt-rt patch enabled. I would like to use native kernel tracing tools such as ftrace, ptrace and eBPF. However I find out they are not fully supported in L4t. For example, according to the tutorial on LWN.net as below.

https://lwn.net/Articles/365835/

Following kernel configurations should be enabled and I can’t turn FUNCTION_GRAPH_TRACER on.

CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE

And for ptrace, I even can’t see man page of ptrace on Xavier. Second, I try to compile pyflame which is depend on ptrace and I failed due to not defining PTRACE_GETREGS.

I wonder if there is any tool which is recommended to trace function and syscall in the kernel. Or maybe how can I use ftrace-related tools to do what I want to do.

#2

The ftrace should be enable by default configure.

You should be able to get below trace node after boot.
sys/kernel/debug/tracing/

#3

I mean part of ftrace functions are enabled. For example, the plugin function-graph is not enabled. Anyway, I solve my issue of identifying hotspot in my process by perf. However, I meet another issue. When I trace the function, I keep seeing undefined symbols in call-graph. My application is pretty simple.

  1. Load images from memory
  2. Make inference with TensorRT
  3. Display result on the monitor

1920×1080 240 KB

I use TF-TRT API to do inference. Those “??” are the symbols which are not defined. I wonder the root cause is from TensorRT or CPython interpreter itself. Following is the perf command for tracing.

perf record --call-graph dwarf -- python3 inference.py

Also, the tensorrt module in python seems not available on arm AArch64 such as Xavier. I can’t import it. Will you recommend me to use C++ API to build my product.

#4

FYI, it looks like “PTRACE_GETREGS” is not implemented on all architectures. It is likely (but not guaranteed) that arm64 does not support this. There are many kernel tracing features and tools which are specific to a given architecture.

#5