Jetson AGX Orin 32GB Redcued GPU Frequency

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

We recently bumped the L4T version that is running on our Jetson AGX Orin 32GB from 35.4.1 to 36.4.4. In that process, we noticed our application ended up running at a lower framerate than what was previously the case. Looking further into the problem, we concluded that our GPU clock frequency is now being set lower than what was previously the case.

Running sudo jetson_clocks –show on the old 35.4.1 image, it outputs

SOC family:tegra234  Machine:Jetson AGX Orin for DSBOARD-AGX
Online CPUs: 0-5
cpu0:  Online=1 Governor=performance MinFreq=1574400 MaxFreq=1574400 CurrentFreq=1574400 IdleStates: WFI=1 c7=1  
cpu1:  Online=1 Governor=performance MinFreq=1574400 MaxFreq=1574400 CurrentFreq=1574400 IdleStates: WFI=1 c7=1  
cpu2:  Online=1 Governor=performance MinFreq=1574400 MaxFreq=1574400 CurrentFreq=1574400 IdleStates: WFI=1 c7=1  
cpu3:  Online=1 Governor=performance MinFreq=1574400 MaxFreq=1574400 CurrentFreq=1574400 IdleStates: WFI=1 c7=1  
cpu4:  Online=1 Governor=performance MinFreq=1574400 MaxFreq=1574400 CurrentFreq=1574400 IdleStates: WFI=1 c7=1  
cpu5:  Online=1 Governor=performance MinFreq=1574400 MaxFreq=1574400 CurrentFreq=1574400 IdleStates: WFI=1 c7=1  
cpu6:  Online=0 Governor=performance MinFreq=1574400 MaxFreq=1574400 CurrentFreq=1574400 IdleStates: WFI=1 c7=1  
cpu7:  Online=0 Governor=performance MinFreq=1574400 MaxFreq=1574400 CurrentFreq=1574400 IdleStates: WFI=1 c7=1  
GPU MinFreq=1300500000 MaxFreq=1300500000 CurrentFreq=1300500000
EMC MinFreq=204000000 MaxFreq=3199000000 CurrentFreq=3199000000 FreqOverride=1
DLA0_CORE:   Online=1 MinFreq=0 MaxFreq=18446744073709551615 CurrentFreq=1600000000
DLA0_FALCON: Online=1 MinFreq=0 MaxFreq=18446744073709551615 CurrentFreq=844800000
DLA1_CORE:   Online=1 MinFreq=0 MaxFreq=18446744073709551615 CurrentFreq=1600000000
DLA1_FALCON: Online=1 MinFreq=0 MaxFreq=18446744073709551615 CurrentFreq=844800000
PVA0_VPS0: Online=1 MinFreq=0 MaxFreq=18446744073709551615 CurrentFreq=1152000000
PVA0_AXI:  Online=1 MinFreq=0 MaxFreq=18446744073709551615 CurrentFreq=832000000
FAN Dynamic Speed control=active hwmon4_pwm1=181

Whereas the newer 36.4.4 image outputs

SOC family:tegra234  Machine:AGX Orin for DSBOARD-AGX
Online CPUs: 0-7
cpu0:  Online=1 Governor=performance MinFreq=729600 MaxFreq=2188800 CurrentFreq=2188800 IdleStates: WFI=1 c7=1  
cpu1:  Online=1 Governor=performance MinFreq=729600 MaxFreq=2188800 CurrentFreq=2188800 IdleStates: WFI=1 c7=1  
cpu2:  Online=1 Governor=performance MinFreq=729600 MaxFreq=2188800 CurrentFreq=2188800 IdleStates: WFI=1 c7=1  
cpu3:  Online=1 Governor=performance MinFreq=729600 MaxFreq=2188800 CurrentFreq=2188800 IdleStates: WFI=1 c7=1  
cpu4:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=2188800 IdleStates: WFI=1 c7=1  
cpu5:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=2035200 IdleStates: WFI=1 c7=1  
cpu6:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=1497600 IdleStates: WFI=1 c7=1  
cpu7:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=1497600 IdleStates: WFI=1 c7=1  
GPU MinFreq=306000000 MaxFreq=930750000 CurrentFreq=612000000
Active GPU TPCs: 7
EMC MinFreq=204000000 MaxFreq=3199000000 CurrentFreq=3199000000 FreqOverride=0
DLA0_CORE:   Online=1 MinFreq=0 MaxFreq=1408000000 CurrentFreq=1408000000
DLA0_FALCON: Online=1 MinFreq=0 MaxFreq=742400000 CurrentFreq=742400000
DLA1_CORE:   Online=1 MinFreq=0 MaxFreq=1408000000 CurrentFreq=1408000000
DLA1_FALCON: Online=1 MinFreq=0 MaxFreq=742400000 CurrentFreq=742400000
PVA0_VPS0: Online=1 MinFreq=0 MaxFreq=1190400000 CurrentFreq=1190400000
PVA0_AXI:  Online=1 MinFreq=0 MaxFreq=857600000 CurrentFreq=857600000
FAN Dynamic Speed Control=nvfancontrol hwmon5_pwm1=204
NV Power Mode: MAXN

We take particular note in comparing the lines (35.4.1):

GPU MinFreq=1300500000 MaxFreq=1300500000 CurrentFreq=1300500000

(36.4.4):

GPU MinFreq=306000000 MaxFreq=930750000 CurrentFreq=612000000

Looking at the technical specifications, it states that our 32GB AGX Orin, quite correctly, should go no higher than 930 MHz, and indeed we appear unable to overclock it. Our question then becomes: why were we previously seemingly able to overclock it to 1.3 GHz? Is there any method one can use to overclock it yet again? On our newer image, we have the power mode set to MAXN, but for the older version, there is no such power mode defined, and running nvpmodel -q returns NVPM WARN: power mode is not set!

Should it be of interest, here is the content of the two /etc/nv_boot_control.conf files

35.4.1:

TNSPEC 3701-501-0004-E.0-1-0-jetson-agx-orin-devkit-
COMPATIBLE_SPEC 3701--0004--1--jetson-agx-orin-devkit-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_EMMC_ONLY false
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

36.4.4:

TNSPEC 3701-500-0004-L.0-1-0-kcc-200-day-mmcblk0p1
COMPATIBLE_SPEC 3701--0004--1--kcc-200-day-
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0
5

Hi hakonaunet,

Are you using the devkit or custom board for AGX Orin?

From the result of nv_boot_control.conf you shared, it seems you used the different board config to flash.

For r36.4.4, could you try using jetson-agx-orin-devkit.conf as board config instead?

1 Like
6

We are using a carrier board provided by Forecr. Making an image with your SDK on 36.4.4 and still using the devkit.conf, and applying minimal Forecr specific patches and device tree binaries, the result still reads

forecsudo r@forecr-desktop:/sys/devices/platform/bus@0/17000000.gpu/devfreq/17000000.gpu$ jetson_clocks --show
SOC family:tegra234  Machine:AGX Orin for DSBOARD-AGX
Online CPUs: 0-7
cpu0:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=1420800 IdleStates: WFI=1 c7=1
cpu1:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=883200 IdleStates: WFI=1 c7=1
cpu2:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=729600 IdleStates: WFI=1 c7=1
cpu3:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=1113600 IdleStates: WFI=1 c7=1
cpu4:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=1497600 IdleStates: WFI=1 c7=1
cpu5:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=1036800 IdleStates: WFI=1 c7=1
cpu6:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=883200 IdleStates: WFI=1 c7=1
cpu7:  Online=1 Governor=schedutil MinFreq=729600 MaxFreq=2188800 CurrentFreq=1036800 IdleStates: WFI=1 c7=1
GPU MinFreq=306000000 MaxFreq=930750000 CurrentFreq=306000000
Active GPU TPCs: 7
EMC MinFreq=204000000 MaxFreq=3199000000 CurrentFreq=665600000 FreqOverride=0
DLA0_CORE:   Online=1 MinFreq=0 MaxFreq=1408000000 CurrentFreq=1408000000
DLA0_FALCON: Online=1 MinFreq=0 MaxFreq=742400000 CurrentFreq=742400000
DLA1_CORE:   Online=1 MinFreq=0 MaxFreq=1408000000 CurrentFreq=1408000000
DLA1_FALCON: Online=1 MinFreq=0 MaxFreq=742400000 CurrentFreq=742400000
PVA0_VPS0: Online=1 MinFreq=0 MaxFreq=1190400000 CurrentFreq=1190400000
PVA0_AXI:  Online=1 MinFreq=0 MaxFreq=857600000 CurrentFreq=857600000
FAN Dynamic Speed Control=nvfancontrol hwmon1_pwm1=178
NV Power Mode: MAXN
 
forecr@forecr-desktop:/sys/devices/platform/bus@0/17000000.gpu/devfreq/17000000.gpu$ sudo cat /etc/nv_boot_control.conf
TNSPEC 3701-500-0004-L.0-1-1-jetson-agx-orin-devkit-
COMPATIBLE_SPEC 3701--0004--1--jetson-agx-orin-devkit-
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0
 
forecr@forecr-desktop:/sys/devices/platform/bus@0/17000000.gpu/devfreq/17000000.gpu$ cat available_frequencies
306000000 408000000 510000000 612000000 714000000 816000000 918000000 930750000

Should it be the case that setting the 32GB AGX Orin’s GPU clocks to 1.3 GHz is no longer available? Writing out the available GPU frequencies on our 35.4.1 image, it reads

cat /sys/devices/gpu.0/devfreq/17000000.ga10b/available_frequencies
306000000 408000000 510000000 612000000 714000000 816000000 918000000 1020000000 1122000000 1224000000 1300500000
7

Does it have the similar design as the devkit?

Why the previous board config you used for r36.4.4 is kcc-200-day-mmcblk0p1.conf and it is jetson-agx-orin-devkit.conf now?
I’m not clear about the design on your board and also what patches/device tree you applied. Please check with your vendor for the custom BSP package and confirm it can boot successfully.

Do you have the AGX Orin devkit to reproduce the issue?

8

Firstly, we are thankful for your continued help.

It should be pretty similar apart from bct-gpio and bct-pinmax.

We switched to using Nvidia’s native jetson-agx-orin-devkit board config just to ensure ourselves, and you, that our custom board config is indifferent in the issue manifesting itself. Everything boots just fine.

Unfortunately, we only have the 64GB version of the AGX Orin devkit, however, we tried interchanging the 64GB AGX Orin in the devkit for an independently sourced 32GB one, assuming it would make no difference. For our testing, we didn’t get to compare the exact patches of L4T major versions 35 and 36, but the most important takeaway from the tests, was that using L4T major version 35 or 36 made no difference for the maximum allowed GPU frequency on our devkit, and it was in both cases 930 MHz.

On our end, it does really appear as if the allowed range of GPU clock rates on the 32GB AGX Orin has been lowered by something some time in between our upgrade from Linux for Tegra’s version 35.4.1 to 36.4.4. Fundamentally, this is the question we are looking to answer. As mentioned, we are fully aware of the data sheet specifying that the 32GB version caps out at 930 MHz, but previously we were able to operate it past this, at 1.3 GHz. If you could provide a definitive answer as to whether any changes have been made on Nvidia’s end to limit the allowed GPU clock ranges on the 32GB version of the Jetson AGX Orin, or what mechanism would explain why the ceiling was previously higher, it would be greatly appreciated.

Very appreciative of your continued assistance and we would welcome any direction on where to focus our troubleshooting efforts going forward.

9

I just get a AGX Orin devkit with 32GB module (SKU4) installed r35.6.0 and the available max GPU frequency is 930750000.

$ cat /etc/nv_tegra_release
# R35 (release), REVISION: 6.0, GCID: , BOARD: t186ref, EABI: aarch64, DATE: 公曆 20廿四年 八月 十三日 週二 十三時54分36秒

nvidia@Jetson:~$ cat /etc/nv_boot_control.conf
TNSPEC 3701-500-0004-G.0-1-1-jetson-agx-orin-devkit-
COMPATIBLE_SPEC 3701--0004--1--jetson-agx-orin-devkit-
TEGRA_LEGACY_UPDATE false
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_EMMC_ONLY false
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

nvidia@Jetson:~$ cat /sys/devices/gpu.0/devfreq/17000000.ga10b/available_frequencies
306000000 408000000 510000000 612000000 714000000 816000000 918000000 930750000

I’m not sure why you get 1300500000 in r35.4.1.

As it has been mentioned in data sheet with 930MHz, we don’t suggest using higher frequency for AGX Orin 32 G SKU4.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.