GPU clock frequency

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hello,
I am running a script that runs N time a CUDA application, and after each call I am displaying the current GPU frequency using the command:

cat /sys/devices/17000000.gv11b/devfreq/17000000.gv11b/cur_freq

I am always getting 1377MHz while the script is running and when it stops I check the frequency again, it falls.
However, I added a new power mode in the “nvpmodel.conf” and I wanted to fix the clock frequency to one of the displayed available frequencies for ex 114750000. I verified that it was fixed to that value and ran again the script, the frequency shown while running is the same 1377Mhz however when the script stops it goes back to 114.75 Mhz.
Could someone help explaining this behavior?

Thank you

3

Hi,

We have tegrastats tool to get the clocks rate and jetson_clocks to fix the clock to maximal.
Have you given it a try?

Thanks.

1 Like
4

Hello,
Yes I tried the jetson_clocks script It didn’t change anything. For the tegrastats, I have tried it today and I am getting the following (launched it in the background with interval 1ms):
07-04-2023 10:49:36 RAM 1441/31011MB (lfb 7043x4MB) SWAP 0/15506MB (cached 0MB) CPU [0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265,0%@2265] EMC_FREQ 0% GR3D_FREQ 0% AUX@41.5C CPU@44.5C thermal@42.7C Tboard@43C AO@41.5C GPU@42C Tdiode@48C PMIC@50C
the GR3D_FREQ is always at 0% and in the case of the Jetson AGX as stated in the doucmentation of tegrastats I cannot get the GPU actual frequency (the @Y). At the same time I didn’t understand what does the % shows although it is written in the documentation that is “Percent of the GR3D that is being used, relative to the current running frequency”.

For reference, here are the outputs of the frequencies:


local@agx-xavier-charles:~/cuda-samples-11.4/Samples/fp16ScalarProduct$ cat /sys/devices/17000000.gv11b/devfreq/17000000.gv11b/cur_freq
1377000000
local@agx-xavier-charles:~/cuda-samples-11.4/Samples/fp16ScalarProduct$ cat /sys/devices/17000000.gv11b/devfreq/17000000.gv11b/min_freq
522750000
local@agx-xavier-charles:~/cuda-samples-11.4/Samples/fp16ScalarProduct$ cat /sys/devices/17000000.gv11b/devfreq/17000000.gv11b/max_freq
522750000

Thank you for your support.

5

Hi,

Have you run it with root?

The 0% represents the utilization ratio.
The clock rate is the value on the right of @.

For example, it is 1377 MHz in the below output.


07-05-2023 05:26:43 ... GR3D_FREQ 0%@1377 ...

Thanks.

1 Like
6

Hello,

Yes thank you it worked using root. However I am still getting 1377Mhz.

I will show you the output of jetson_clocks:

SOC family:tegra194  Machine:Jetson-AGX
Online CPUs: 0-3
cpu0: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=0 c6=0
cpu1: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=0 c6=0
cpu2: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=0 c6=0
cpu3: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=0 c6=0
cpu4: Online=0 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=0 c6=0
cpu5: Online=0 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=0 c6=0
cpu6: Online=0 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=0 c6=0
cpu7: Online=0 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=0 c6=0
GPU MinFreq=675750000 MaxFreq=675750000 CurrentFreq=1377000000
EMC MinFreq=204000000 MaxFreq=1331200000 CurrentFreq=1331200000 FreqOverride=0
DLA0_CORE MinFreq=0 MaxFreq=752250000 CurrentFreq=752250000
DLA0_FALCON MinFreq=0 MaxFreq=450000000 CurrentFreq=450000000
DLA1_CORE MinFreq=0 MaxFreq=752250000 CurrentFreq=752250000
DLA1_FALCON MinFreq=0 MaxFreq=450000000 CurrentFreq=450000000
PVA0_VPS0 MinFreq=0 MaxFreq=550400000 CurrentFreq=550400000
PVA0_VPS1 MinFreq=0 MaxFreq=550400000 CurrentFreq=550400000
PVA0_AXI MinFreq=0 MaxFreq=385950000 CurrentFreq=385950000
PVA1_VPS0 MinFreq=0 MaxFreq=550400000 CurrentFreq=550400000
PVA1_VPS1 MinFreq=0 MaxFreq=550400000 CurrentFreq=550400000
PVA1_AXI MinFreq=0 MaxFreq=385950000 CurrentFreq=385950000
CVNAS MinFreq=0 MaxFreq=716800000 CurrentFreq=716800000
FAN Dynamic Speed control=active hwmon2_pwm=0
NV Power Mode: MODE_15W

and here is the ouptput of tegrastats:

07-05-2023 09:13:09 RAM 1502/31011MB (lfb 6898x4MB) SWAP 0/15506MB (cached 0MB) CPU [66%@1190,50%@1189,100%@1190,0%@1189,off,off,off,off] EMC_FREQ 0%@1331 GR3D_FREQ 2%@1377 VIC_FREQ 115 APE 150 AUX@44C CPU@46C thermal@44.9C Tboard@45C AO@44C GPU@45C Tdiode@50.5C PMIC@50C GPU 1087mW/1087mW CPU 776mW/776mW SOC 1243mW/1243mW CV 0mW/0mW VDDRQ 465mW/465mW SYS5V 2009mW/2009mW

I believe maybe there is a frequency scaling that is being done on the GPU (governor…)

Update: I was reading on this forum and I have found this:

There is a parameter which was set to 1377MHz, so I change it, and the frequency changed afterwards. It is weird that this parameter is also used.

Thanks again

7

Hi,

Could you share more about your use case?

After running jetson_clocks, the GPU clock will be locked to the maximum to avoid the overhead of dynamic frequency.
If you want the clock rate to drop when idle, please set it with only nvpmodel. Not lock with jetson_clocks.

Thanks,

1 Like
8

Hello,

This is my case:
I want to lock the GPU frequency to a certain frequency. So I used the documentation to check how to do it. So I did as the documentation says by making the minimum frequency equal to maximum frequency using nvpmodel. I applied it however I checked the frequency it was always locked to 1337MHz without using the jetson_clocks, eventhough the min and max frequencies were below 1337MHz.

ex:

GPU MinFreq=675750000 MaxFreq=675750000 CurrentFreq=1377000000

That’s it.

Thank you

9

Hi,

Thanks for the clarification.
Could you share the nvpmodel configuration so we can reproduce this in our environment?
Thanks.

1 Like
10

Hello,

Please find attached the configuration file, as well as the output from the jetson_clock --show after a restart

local@agx-xavier-charles:~$ sudo /usr/bin/jetson_clocks --show
SOC family:tegra194  Machine:Jetson-AGX
Online CPUs: 0-7
cpu0: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1958400 IdleStates: C1=1 c6=1
cpu1: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=1958400 IdleStates: C1=1 c6=1
cpu2: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu3: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu4: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2035200 IdleStates: C1=1 c6=1
cpu5: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu6: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu7: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
GPU MinFreq=522750000 MaxFreq=522750000 CurrentFreq=522750000
EMC MinFreq=204000000 MaxFreq=1331200000 CurrentFreq=1331200000 FreqOverride=0
DLA0_CORE MinFreq=0 MaxFreq=1395200000 CurrentFreq=1395200000
DLA0_FALCON MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
DLA1_CORE MinFreq=0 MaxFreq=1395200000 CurrentFreq=1395200000
DLA1_FALCON MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
PVA0_VPS0 MinFreq=0 MaxFreq=1088000000 CurrentFreq=1088000000
PVA0_VPS1 MinFreq=0 MaxFreq=1088000000 CurrentFreq=1088000000
PVA0_AXI MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
PVA1_VPS0 MinFreq=0 MaxFreq=1088000000 CurrentFreq=1088000000
PVA1_VPS1 MinFreq=0 MaxFreq=1088000000 CurrentFreq=1088000000
PVA1_AXI MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
CVNAS MinFreq=0 MaxFreq=1356800000 CurrentFreq=1356800000
FAN Dynamic Speed control=active hwmon2_pwm=0
NV Power Mode: Ant

jetson_clock after running a kernel:

SOC family:tegra194  Machine:Jetson-AGX
Online CPUs: 0-7
cpu0: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu1: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu2: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu3: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu4: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu5: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu6: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
cpu7: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=1 c6=1
GPU MinFreq=522750000 MaxFreq=522750000 CurrentFreq=1377000000
EMC MinFreq=204000000 MaxFreq=1331200000 CurrentFreq=1331200000 FreqOverride=0
DLA0_CORE MinFreq=0 MaxFreq=1395200000 CurrentFreq=1395200000
DLA0_FALCON MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
DLA1_CORE MinFreq=0 MaxFreq=1395200000 CurrentFreq=1395200000
DLA1_FALCON MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
PVA0_VPS0 MinFreq=0 MaxFreq=1088000000 CurrentFreq=1088000000
PVA0_VPS1 MinFreq=0 MaxFreq=1088000000 CurrentFreq=1088000000
PVA0_AXI MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
PVA1_VPS0 MinFreq=0 MaxFreq=1088000000 CurrentFreq=1088000000
PVA1_VPS1 MinFreq=0 MaxFreq=1088000000 CurrentFreq=1088000000
PVA1_AXI MinFreq=0 MaxFreq=844800000 CurrentFreq=844800000
CVNAS MinFreq=0 MaxFreq=1356800000 CurrentFreq=1356800000
FAN Dynamic Speed control=active hwmon2_pwm=0
NV Power Mode: Ant

tegrastat output after first run:

07-13-2023 09:03:45 RAM 1547/31011MB (lfb 7082x4MB) SWAP 0/15506MB (cached 0MB) CPU [4%@1189,2%@1190,5%@1187,3%@1188,3%@1193,6%@1189,9%@1190,7%@1188] EMC_FREQ 0%@665 GR3D_FREQ 0%@1377 VIC_FREQ 115 APE 150 AUX@43C CPU@45.5C thermal@44.5C Tboard@45C AO@43.5C GPU@45.5C Tdiode@50C PMIC@50C GPU 0mW/0mW CPU 777mW/777mW SOC 777mW/777mW CV 0mW/0mW VDDRQ 311mW/311mW SYS5V 1326mW/1326mW

nvpmodel -q output:

local@agx-xavier-charles:~$ sudo nvpmodel --q
NV Power Mode: Ant
8

nvpmodel.conf (16.5 KB)

Thank you for your support.

11

Hi,

Thanks for the file.

We are going to reproduce this issue internally.
Will share more info with you later.

Thanks.

1 Like
13

Hi,

We try the config file but were NOT able to reproduce this issue.

Setup

$ sudo nvpmodel -f nvpmodel.conf
$ sudo nvpmodel -m 8
$ sudo jetson_clocks --show
SOC family:tegra194  Machine:Jetson-AGX
Online CPUs: 0-7
...
GPU MinFreq=522750000 MaxFreq=522750000 CurrentFreq=522750000
...

Test

Run both fp16ScalarProduct or some GPU-intensive sample (ex. TensorRT)

$ sudo tegrastats
07-18-2023 05:24:23 ... GR3D_FREQ 99%@522 ...

$ sudo jetson_clocks --show
SOC family:tegra194  Machine:Jetson-AGX
...
GPU MinFreq=522750000 MaxFreq=522750000 CurrentFreq=522750000
...
NV Power Mode: Ant

The clock rate is still 522MHz in our environment.
Do we miss anything?

Thanks.

1 Like
14

Hello,

Thank for sharing your result. Well I don’t believe you are missing something I did the same. Are you running on the latest Jetpack version?

15

Hi,

Yes, we try it on JetPack 5.1.1.

Actually, we don’t reboot the system after applying the power mode.
Will this cause something different?

Thanks.

1 Like
16

Hello,

No, I tried the settings with and without the restart. It is the same. Maybe then I need to update to the latest Jetpack since I am running on the version 5.0.2-b231.

Thanks a lot for your support

19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.