Issue of computing speed are slow at first and then accelerate

[timelim]

We run signal processing programmes based on ROS. We calculate the execution time using the Chrono library that comes with C++11. The signal processing is measured in frames, and each frame performs exactly the same processing flow.

During the processing task, the performance is low and unstable at the beginning of the programme. After a few frames have been processed (always takes about 1.8 seconds), the performance starts to increase and time consuming going to be stabily in each frame (100ms/frame → 70ms/frame).

However, for the same programme and data, the time taken for each frame is always approximately in same when we run it on another servers which equipped with Intel processors.

The uneven time consuming of each frame is a fatal problem for my task, so I would like to know how I can solve this problem.

For more information:

1. I noticed that the cpu usage is 100% and power is 18w when processing is solw, but around 40%, 9W when the speed increased.
2. The power mode is “0:MAXN”.

[AastaLLL]

Hi,

Does the task only use CPU for calculation?
Could you try to fix the processor clock to the maximum?

$ sudo jetson_clocks

Jetson by default uses the dynamic clock rate.
It’s possible that the performance is lower at first if the clock rate starts from low.

Thanks.

[timelim]

Thanks for your reply !
It’s right, our task only use CPU for calculation.
I tried the command but it seems like nothing was changed. The processing still slow at first and then faster.
just like:

Time Consuming
100 ms
^    $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ 
|                                   $                                            
|                                   $                                            
|                                   $                                            
|                                   $                                                                             
|                                    $                                           
|                                    $                                           
|                                    $                                           
|                                    B                                           
|                                     $                                          
|                                     $                                          
|                                     $                                                       
|70ms                                 .   $..$       $$                          
|                                      @       $..@    .$                        
|                                               .        $$$$$$ $$$               
|                                                             $   $$$$$$$$$$$$$$  
|
|
|
0ms -----------------------------------------------------------------------------> frames

Moreover, we found the value of EMC usage in jtop tools always be 2~4% when the performance is low, but the EMC usage keep around 7% when processing get faster.

[AastaLLL]

Hi,

jetson_clocks will also set the EMC clock to the maximum.
Could you check if the EMC clock remains the same in both scenarios?

$ cat /sys/kernel/debug/bpmp/debug/clk/emc/rate

Thanks.

[timelim]

Here’s the

$ sudo jetson_clocks

$ sudo jetson_clocks --show
SOC family:tegra234  Machine:NVIDIA Orin Jetson-Small Developer Kit
Online CPUs: 0-11
cpu0: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu1: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu10: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu11: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu2: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu3: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu4: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu5: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu6: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu7: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu8: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
cpu9: Online=1 Governor=schedutil MinFreq=2201600 MaxFreq=2201600 CurrentFreq=2201600 IdleStates: WFI=0 c7=0 
GPU MinFreq=1300500000 MaxFreq=1300500000 CurrentFreq=1300500000
EMC MinFreq=204000000 MaxFreq=3199000000 CurrentFreq=3199000000 FreqOverride=1
DLA0_CORE MinFreq=0 MaxFreq=1536000000 CurrentFreq=1536000000
DLA0_FALCON MinFreq=0 MaxFreq=832000000 CurrentFreq=832000000
DLA1_CORE MinFreq=0 MaxFreq=1536000000 CurrentFreq=1536000000
DLA1_FALCON MinFreq=0 MaxFreq=832000000 CurrentFreq=832000000
PVA0_VPS0 MinFreq=0 MaxFreq=1152000000 CurrentFreq=1152000000
PVA0_AXI MinFreq=0 MaxFreq=832000000 CurrentFreq=832000000
FAN Dynamic Speed control=active hwmon3_pwm=88
NV Power Mode: MAXN

$ sudo cat /sys/kernel/debug/bpmp/debug/clk/emc/rate
3199000000

In jtop, EMC stays at 3.2GHz and All CPUs stay at 2.2GHz, but the problem is still there. The CPU Usage value and the CPU CV Power value will go down while processing will be faster.

[AastaLLL]

Hi,

Is this issue can be reproduced without ROS?
If yes, could you share a reproducible app with us?

We want to have a local repo to get more information.

Thanks.

[timelim]

We will do the testing without ROS and thanks for your help!