EMC clock stuck at 204 MHz

Shadowmind · March 26, 2021, 5:01pm

We are building a Yocto build of Linux from Jetpack 4.4.2. I’ve noticed the EMC clock never changes from 204 MHz. Our application is pushing the EMC to 70-75% but the clock never changes.

I have run:

   jetson_clocks
   jetson_clocks --show

This is the result:

   root@maza2:~# jetson_clocks --show
   SOC family:tegra210  Machine:Appareo XXX
   Online CPUs: 0-1
   CPU Cluster Switching: Disabled
   cpu0: Online=1 Governor=schedutil MinFreq=921600 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=0 c7=0 
   cpu1: Online=1 Governor=schedutil MinFreq=921600 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=0 c7=0 
   cpu2: Online=0 Governor=schedutil MinFreq=921600 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=0 c7=0 
   cpu3: Online=0 Governor=schedutil MinFreq=921600 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=0 c7=0 
   GPU MinFreq=614400000 MaxFreq=614400000 CurrentFreq=614400000
   EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=204000000 FreqOverride=1
   Can't access Fan!
   NV Power Mode: MAXN

I have also tried running commands to directly set the clock rate, but it always stays at 204 MHz.
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/power_management_nano.html#wwpID0E0SL0HA
I have run stress-ng and the clock never changes. Is there a setting somewhere else that I am missing

ShaneCCC · March 29, 2021, 3:44am

To set static EMC frequency

•Enter the commands:

$ echo <emc_freq> > /sys/kernel/debug/clk/override.emc/clk_update_rate

$ echo 1 > /sys/kernel/debug/clk/override.emc/clk_state

Shadowmind · March 29, 2021, 4:24pm

here is the output of Tegrastats prior to running the above command:

RAM 681/3965MB (lfb 601x4MB) IRAM 0/252kB(lfb 252kB) CPU [8%@102,1%@102,5%@102,1%@102] EMC_FREQ 0%@204 GR3D_FREQ 0%@76 APE 204 PLL@29.5C CPU@29.5C PMIC@100C GPU@31.5C AO@37.5C thermal@30.5C POM_5V_GPU 1088/1138 POM_5V_IN 0/0 POM_5V_CPU 120/152

I then run the following commands:

echo 1600000000 > /sys/kernel/debug/clk/override.emc/clk_update_rate
echo 1 > /sys/kernel/debug/clk/override.emc/clk_state

Tegrastats then reports:

RAM 680/3965MB (lfb 602x4MB) IRAM 0/252kB(lfb 252kB) CPU [7%@102,2%@102,2%@102,3%@102] EMC_FREQ 0%@204 GR3D_FREQ 0%@76 APE 204 PLL@29.5C CPU@30C PMIC@100C GPU@31.5C AO@37.5C thermal@31C POM_5V_GPU 1088/1088 POM_5V_IN 0/0 POM_5V_CPU 120/131

There are no additional messages in DMESG output.

ShaneCCC · March 29, 2021, 6:15pm

What the frequency for “jetson_clocks --show”

Shadowmind · March 30, 2021, 8:48pm

Below is from a fresh boot:

root@maza2:~# jetson_clocks --show
SOC family:tegra210  Machine:Appareo Maza2
Online CPUs: 0-1
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=1 c7=1 
cpu1: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=1 c7=1 
cpu2: Online=0 Governor=schedutil MinFreq=102000 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=1 c7=1 
cpu3: Online=0 Governor=schedutil MinFreq=102000 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=1 c7=1 
GPU MinFreq=76800000 MaxFreq=614400000 CurrentFreq=76800000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=204000000 FreqOverride=0
Can't access Fan!
NV Power Mode: 5W

After running

echo 1600000000 > /sys/kernel/debug/clk/override.emc/clk_update_rate
echo 1 > /sys/kernel/debug/clk/override.emc/clk_state

I get the following. So I see the FreqOverride is set. I don’t see any error messages in dmesg or /var/log/messages.

root@maza2:~# jetson_clocks --show
SOC family:tegra210  Machine:Appareo Maza2
Online CPUs: 0-1
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=921600 CurrentFreq=825600 IdleStates: WFI=1 c7=1 
cpu1: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=1 c7=1 
cpu2: Online=0 Governor=schedutil MinFreq=102000 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=1 c7=1 
cpu3: Online=0 Governor=schedutil MinFreq=102000 MaxFreq=921600 CurrentFreq=921600 IdleStates: WFI=1 c7=1 
GPU MinFreq=76800000 MaxFreq=614400000 CurrentFreq=76800000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=204000000 FreqOverride=1
Can't access Fan!
NV Power Mode: 5W

ShaneCCC · March 31, 2021, 4:05am

Below is from my Nano. It’s J4.5

nvidia@nvidia-desktop:~$ sudo su
[sudo] password for nvidia:
root@nvidia-desktop:/home/nvidia# echo 1600000000 > /sys/kernel/debug/clk/override.emc/clk_update_rate
root@nvidia-desktop:/home/nvidia# echo 1 > /sys/kernel/debug/clk/override.emc/clk_state
root@nvidia-desktop:/home/nvidia# jetson_clocks --show
SOC family:tegra210  Machine:NVIDIA Jetson Nano Developer Kit
Online CPUs: 0-3
cpu0: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=614400 IdleStates: WFI=1 c7=1
cpu1: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=825600 IdleStates: WFI=1 c7=1
cpu2: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=825600 IdleStates: WFI=1 c7=1
cpu3: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=825600 IdleStates: WFI=1 c7=1
GPU MinFreq=76800000 MaxFreq=921600000 CurrentFreq=76800000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=1
Fan: PWM=0
NV Power Mode: MAXN
root@nvidia-desktop:/home/nvidia# tegrastats
RAM 1158/3956MB (lfb 451x4MB) SWAP 0/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@102,4%@102,2%@102,4%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@24.5C CPU@27C PMIC@100C GPU@26.5C AO@33C thermal@26.75C POM_5V_IN 1892/1892 POM_5V_GPU 0/0 POM_5V_CPU 123/123
RAM 1158/3956MB (lfb 451x4MB) SWAP 0/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@102,2%@102,1%@102,2%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@24.5C CPU@27C PMIC@100C GPU@26.5C AO@32.5C thermal@26.75C POM_5V_IN 1934/1913 POM_5V_GPU 0/0 POM_5V_CPU 123/123
RAM 1158/3956MB (lfb 451x4MB) SWAP 0/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@102,2%@102,0%@102,4%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@24.5C CPU@27C PMIC@100C GPU@26.5C AO@32.5C thermal@26.75C POM_5V_IN 1934/1920 POM_5V_GPU 0/0 POM_5V_CPU 123/123
RAM 1158/3956MB (lfb 451x4MB) SWAP 0/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@204,2%@204,2%@204,2%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@24.5C CPU@27C PMIC@100C GPU@26.5C AO@32.5C thermal@26.5C POM_5V_IN 1892/1913 POM_5V_GPU 0/0 POM_5V_CPU 123/123
RAM 1158/3956MB (lfb 451x4MB) SWAP 0/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [8%@102,1%@102,0%@102,3%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@25C CPU@27C PMIC@100C GPU@27C AO@32.5C thermal@26.5C POM_5V_IN 1934/1917 POM_5V_GPU 0/0 POM_5V_CPU 123/123

ShaneCCC · March 31, 2021, 4:20am

Looks like the default clock are 1600. J4.4 without problem too.

Last login: Tue Nov  3 16:32:57 2020 from 192.168.55.100
nvidia@nvidia-desktop:~$ sudo jetson_clocks --show
[sudo] password for nvidia:
SOC family:tegra210  Machine:NVIDIA Jetson Nano Developer Kit
Online CPUs: 0-3
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=921600 IdleStates: WFI=1 c7=1
cpu1: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=825600 IdleStates: WFI=1 c7=1
cpu2: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=1036800 IdleStates: WFI=1 c7=1
cpu3: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=710400 IdleStates: WFI=1 c7=1
GPU MinFreq=76800000 MaxFreq=921600000 CurrentFreq=76800000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=0
Fan: speed=0
NV Power Mode: MAXN
nvidia@nvidia-desktop:~$ cat /etc/nv_tegra_release
# R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t210ref, EABI: aarch64, DATE: Fri Jun 26 04:38:25 UTC 2020

Shadowmind · March 31, 2021, 7:59pm

This is an odd one. We recently switched Linux to use the device tree in /boot. We are using rauc to perform an in field upgrade. But when we update the Linux DTB partition, Linux doesn’t actually use that device tree. It appears Linux is using the tegraboot/cboot devicetree. I can’t find a way to force Linux to use the updated devicetree. So we had no way of updating the device tree in the field. Therefore we switched u-boot to load the device tree from /boot via the extlinux.conf file.

# Generic Distro Configuration file generated by OpenEmbedded
LABEL Appareo Systems Distro
	KERNEL ../Image
	FDT ../tegra210-maza2.dtb
	INITRD ../initrd
	APPEND ${cbootargs} root=/dev/mmcblk${devnum}p${distro_bootpart} rw rootwait ${bootargs}

If I boot a unit using that method, the EMC clock is fixed at 204 MHz. If I remove the FDT line in extlinux.conf, then the device boots and the EMC clock is at 1600 MHz. I don’t understand why that is the case.

I thought that the cmdline might be different between the two boots, but it is the same:

tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,2 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 tegra_fbmem=0x140000@0x92cb0000 is_hdmi_initialised=1  console=ttyS0,115200 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1 root=/dev/mmcblk0p1 rw rootwait rauc.slot=A

Any ideas what is going on here? Is cboot modifying the devicetree before passing it to the kernel? The only difference is how the kernel obtains the device tree. I’m attaching the boot logs for both boots.

bad_clk.txt (69.4 KB)
good_clk.txt (68.5 KB)

ShaneCCC · April 1, 2021, 3:54am

Did you have any modify for the device tree?
Have a try to update the device tree by replace the dtb in…/Linux_for_Tegra/kernel/dtb/ and issue below command.

sudo ./flash.sh -r -k DTB jetson-nano-xxx mmcblk0p1

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html#

Shadowmind · April 1, 2021, 3:52pm

My understanding is flash.sh and the associated tools are X86 artifacts, at least looking at the binaries they are x86 ELF files. In my product, a firmware update will be send over a cellular connection to the device. The device must be able to perform the firmware update, including DTB updates without human intervention. Is this not possible?

ShaneCCC · April 5, 2021, 3:41pm

For your case I would suggest to dump the DTB from the /proc/device-tree by dtc like below command.
Then modify the extracted_proc.dts to add your context and using dtc to compile it to DTB and modify the extlinux.cof to apply it.

sudo dtc -I fs -O dts -o extracted_proc.dts /proc/device-tree

Topic		Replies	Views
CPU cores always run at highest frequency... why? Jetson Nano kernel	26	1408	October 18, 2021
Not able to set EMC clock to maximum frequency Jetson TX1	4	1422	October 18, 2021
Jetson nano black screen after fresh install Jetson Nano reflash	7	286	May 21, 2024
Jetson Nano booting at custom main board Jetson Nano	13	1529	October 14, 2021
Kernel error occurs when inserting SD card Jetson Nano kernel	16	2395	October 15, 2021
Jetson TK1 output issue Jetson TK1	23	1017	January 3, 2024
Facing problem while flashing the eMMC on the Jetson Nano production ready module (not the Dev Kit) Jetson Nano	28	3517	October 14, 2021
how to reduce cboot&&tegraboot time Jetson Nano	10	1438	October 18, 2021
Jetson Nano boot problem Jetson Nano boot , kernel , ubuntu	5	802	August 6, 2023
kernel-4.4/drivers/platform/tegra/mc/isomgr.c: isomgr_init() fails to initialize Jetson TX2	42	6665	October 18, 2021

EMC clock stuck at 204 MHz

Related topics