Recommended kernel tracing tool

I am using L4t kernel with preempt-rt patch enabled. I would like to use native kernel tracing tools such as ftrace, ptrace and eBPF. However I find out they are not fully supported in L4t. For example, according to the tutorial on as below.

Following kernel configurations should be enabled and I can’t turn FUNCTION_GRAPH_TRACER on.


And for ptrace, I even can’t see man page of ptrace on Xavier. Second, I try to compile pyflame which is depend on ptrace and I failed due to not defining PTRACE_GETREGS.

I wonder if there is any tool which is recommended to trace function and syscall in the kernel. Or maybe how can I use ftrace-related tools to do what I want to do.

The ftrace should be enable by default configure.

You should be able to get below trace node after boot.

I mean part of ftrace functions are enabled. For example, the plugin function-graph is not enabled. Anyway, I solve my issue of identifying hotspot in my process by perf. However, I meet another issue. When I trace the function, I keep seeing undefined symbols in call-graph. My application is pretty simple.

  1. Load images from memory
  2. Make inference with TensorRT
  3. Display result on the monitor

I use TF-TRT API to do inference. Those “??” are the symbols which are not defined. I wonder the root cause is from TensorRT or CPython interpreter itself. Following is the perf command for tracing.

perf record --call-graph dwarf -- python3

Also, the tensorrt module in python seems not available on arm AArch64 such as Xavier. I can’t import it. Will you recommend me to use C++ API to build my product.

FYI, it looks like “PTRACE_GETREGS” is not implemented on all architectures. It is likely (but not guaranteed) that arm64 does not support this. There are many kernel tracing features and tools which are specific to a given architecture.