Orin-x cannot get right call graph via valgrind or perf

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
1

Hi,
I want use valgrind or perf to test the performance of my program. But the two did not get right call graph.

  • valgrind:
#include <stdio.h>
#include <unistd.h>

void func_888()
{
    printf("hello world\n");
    return;
}

void func_666()
{
        printf("你好 世界\n");
        return;
}

int main(int argc, char *argv[])
{
    while(1)
    {
        func_888();
        printf("blabla...\n");
        func_666();
        usleep(10);
    }
    return 0;
}

Compile with follow command: gcc test.c -O0 -g
Then use vallgrind to get log: valgrind --tool=callgrind ./a.out
KCachegrind to view the result.
The result of x86_64 is:

x86
x862020×1179 87.1 KB

The result of orin is:
orin
orin2019×1664 90.1 KB

It did not get the right call graph and call times of func_888() and func_666() on orin.

  • perf
#include <unistd.h>

void bar(){
//   usleep(40*1000);
  /* do something here */
  for(int i=0;i< 4000;i++)
  {

  }
}

void foo(){
//   usleep(60*1000);
  for(int i=0;i< 5700;i++)
  {

  }
  bar();
}

void do_main() {
  foo();
}

int main(int argc,char** argv){
    while(1)
    {
        do_main();
    }
}

compile the program with follow command: gcc test.c -O0 -g
execute the program and use perf to analyse.

sudo perf record -e cpu-clock -g -p 28591
perf script -i perf.data &> perf.unfold
./stackcollapse-perf.pl perf.unfold &> perf.folded
./flamegraph.pl perf.folded > perf.svg

the result of x86_64:

image
image1206×192 7.32 KB

the result of orin:
image
image1199×195 7.51 KB

It can not get the user space stack symbol on orin.

orin system:
Linux tegra-ubuntu 5.10.120-rt70-tegra #7 SMP PREEMPT RT Wed Jun 28 14:23:08 CST 2023 aarch64 aarch64 aarch64 GNU/Linux
orin valgrind version: valgrind-3.21.0
orin perf version 5.10.120

x86_64 system:
Linux ubuntu 5.15.0-72-generic #79~20.04.1-Ubuntu SMP Thu Apr 20 22:12:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
x86_64 valgrind version: valgrind-3.21.0
x86_64 perf version 5.15.98

3

Hi,
Looks like you are running RT kernel. May try default kernel image and see if it works as expected. And may try and compare x86 system in K5.10, or other embedded systems.

We don’t have additional change for valgrind. It is expected to work in default behavior. Would see if other users have seen similar behavior and can share experience.

4

Hi, DaneLLL:
I have dealed with the problem of ‘perf’. It seems because of the cross compile environment. I did not install the libelf-dev on my cross compile environment, it is essential for user space symbol.
image

As for the valgrind, we did not figure out it neither.

5

You can install libelf-dev on the Jetson, and then clone it. If you have all of your dev content on a Jetson, and then use the raw clone (while loopback mounted) as your sysroot during cross compile, then you have everything you need in user space.

6

@linuxdev yeah, we have successfully deal with it (perf) on drive os, but the valgrind stumped us now.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.