I am using L4t kernel with preempt-rt patch enabled. I would like to use native kernel tracing tools such as ftrace, ptrace and eBPF. However I find out they are not fully supported in L4t. For example, according to the tutorial on LWN.net as below.
https://lwn.net/Articles/365835/
Following kernel configurations should be enabled and I can’t turn FUNCTION_GRAPH_TRACER on.
CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE
And for ptrace, I even can’t see man page of ptrace on Xavier. Second, I try to compile pyflame which is depend on ptrace and I failed due to not defining PTRACE_GETREGS.
I wonder if there is any tool which is recommended to trace function and syscall in the kernel. Or maybe how can I use ftrace-related tools to do what I want to do.
The ftrace should be enable by default configure.
You should be able to get below trace node after boot.
sys/kernel/debug/tracing/
I mean part of ftrace functions are enabled. For example, the plugin function-graph is not enabled. Anyway, I solve my issue of identifying hotspot in my process by perf. However, I meet another issue. When I trace the function, I keep seeing undefined symbols in call-graph. My application is pretty simple.
- Load images from memory
- Make inference with TensorRT
- Display result on the monitor
I use TF-TRT API to do inference. Those “??” are the symbols which are not defined. I wonder the root cause is from TensorRT or CPython interpreter itself. Following is the perf command for tracing.
perf record --call-graph dwarf -- python3 inference.py
Also, the tensorrt module in python seems not available on arm AArch64 such as Xavier. I can’t import it. Will you recommend me to use C++ API to build my product.
FYI, it looks like “PTRACE_GETREGS” is not implemented on all architectures. It is likely (but not guaranteed) that arm64 does not support this. There are many kernel tracing features and tools which are specific to a given architecture.
