SC7休眠後 不明原因自動離開SC7模式

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

SC7休眠後 不明原因自動離開SC7模式 …導致後續按下power Key 離開休眠模式時…會甦醒失敗

但有時又正常 不會自動離開SC7模式,按下power Key 離開休眠模式…可正常甦醒

root@Jetson:/home/nvidia# [  620.547434] Trying to unregister non-registered hwtime source
?歊lk_mach_suspend_early
clk_mach_suspend_early done
suspended vdd_core @ 950000uV
跴pe entering sc7
歊lk_mach_suspend
clk_mach_suspend done
fmon_suspend done
t234_adc_suspend done
WAKE_MASK[31:0]  = 0x21000100
WAKE_MASK[63:32] = 0x0
WAKE_MASK[95:64] = 0x1ff200
TIER0[31:0]      = 0x0
TIER0[63:32]     = 0x0
TIER0[95:64]     = 0x0
TIER1[31:0]      = 0x0
TIER1[63:32]     = 0x0
TIER1[95:64]     = 0x0
TIER2[31:0]      = 0x21000100
TIER2[63:32]     = 0x0
TIER2[95:64]     = 0xff200
[0649.295] I> MB1 (version: 1.2.0.0-t234-54845784-9613e64d)
[0649.301] I> t234-A01-0-Silicon (0x12347) Prod
[0649.307] I> Boot-mode : SC7 Exit
[0649.311] I> Entry timestamp: 0x00000000
[0649.316] I> last_boot_error: 0x0
[0649.320] I> BR-BCT: preprod_dev_sign: 0
[0649.325] I> rst_source: 0xc, rst_level: 0x3
[0649.330] I> Task: SE error check
[0649.334] I> Task: Enable SLCG
[0649.338] I> Task: CRC check
[0649.341] I> Skip FUSE records CRC check as records_integrity fuse is not burned
[0649.350] I> Task: Crypto init
[0649.354] I> Task: NVRNG health check
[0649.358] I> NVRNG: Health check success
[0649.363] I> Task: MSS Bandwidth limiter settings for iGPU clients
[0649.370] I> Task: Verify SDRAM params
[0649.376] I> Task: MSS Encrypt status check
[0649.381] I> Task: NV_SC7 integrity check
[0649.390] I> Task: Initialize SOC Therm
[0649.395] I> Task: Program NV master stream id
[0649.400] I> Task: Verify boot mode
[0649.407] I> Task: Alias fuses
[0649.412] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0649.420] I> Task: Print SKU type
[0649.424] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x00000038
[0649.430] I> FUSE_OPT_GPC_DISABLE = 0x00000000
[0649.436] I> FUSE_OPT_TPC_DISABLE = 0x00000002
[0649.441] I> FUSE_OPT_DLA_DISABLE = 0x00000000
[0649.446] I> FUSE_OPT_PVA_DISABLE = 0x00000000
[0649.452] I> FUSE_OPT_NVENC_DISABLE = 0x00000000
[0649.457] I> FUSE_OPT_NVDEC_DISABLE = 0x00000000
[0649.462] I> FUSE_OPT_FSI_DISABLE = 0x00000000
[0649.468] I> FUSE_OPT_EMC_DISABLE = 0x00000000
[0649.473] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0649.478] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0649.483] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0649.488] I> FUSE_SKU_INFO_0 = 0xd2
[0649.493] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS
[0649.498] I> FUSE_PACKAGE_INFO_0 = 0x2
[0649.502] I> SKU: Prod
[0649.506] I> Task: Boost clocks
[0649.509] I> Initializing PLLC2 for AXI_CBB.
[0649.515] I> AXI_CBB : src = 35, divisor = 0
[0649.520] I> Task: Voltage monitor
[0649.524] I> VMON: Vmon re-calibration and fine tuning done
[0649.531] I> Task: TSC init
[0649.535] I> Task: Thermal check
[0649.539] I> max_chip_limit = 105
[0649.543] I> min_chip_limit = -28
[0649.547] I> max temp read = 32
[0649.551] I> min temp read = 31
[0649.555] I> Task: Update FSI SCR with thermal fuse data
[0649.561] I> Task: Enable WDT 5th expiry
[0649.566] I> Task: I2C register
[0649.570] I> Task: Set I2C bus freq
[0649.574] I> Task: Reset FSI
[0649.578] I> Task: Pinmux init
[0649.589] I> Task: Prod config init
[0649.594] I> Task: Pad voltage init
[0649.598] I> Task: Prod init
[0649.602] I> Task: Common rail init
[0649.607] I> DONE: Thermal config
[0649.613] W> DEVICE_PROD: module = 13, instance = 4 not found in device prod.
[0649.623] I> DONE: SOC rail config
[0649.629] W> PMIC_CONFIG: Rail: MEMIO rail config not found in MB1 BCT.
[0649.636] I> DONE: MEMIO rail config
[0649.641] I> DONE: GPU rail info
[0649.645] I> DONE: CV rail info
[0649.649] I> Task: Misc. board config
[0649.654] I> PMIC_CONFIG: Platform config not found in MB1 BCT.
[0649.661] I> Task: Enable clock-mon
[0649.677] I> FMON: Fmon re-programming done
[0649.682] I> Task: CCPLEX IST init
[0649.687] I> Task: CPU WP0
[0649.691] I> Loading MCE
[0649.708] I> Sending WP0 mailbox command to PSC
[0649.718] I> Task: MB1 fixed firewalls
[0649.744] I> Task: BPMP fw ast config
[0649.748] I> Task: Load nvdec-fw
[0649.766] I> Task: Program TSEC carveout
[0649.770] I> TSEC-FW load not enabled
[0649.775] I> Task: GPIO interrupt map
[0649.780] I> Task: Unpowergate AOPG CAN
[0649.785] I> Task: Misc NV security settings
[0649.790] I> NVDEC sticky bits programming done
[0649.795] I> Successfully powergated NVDEC
[0649.800] I> Task: Disable/Reload WDT
[0649.805] I> Task: Program BPMP-IPC carveouts
[0649.810] I> Program IPC carveouts
[0649.816] I> SLCG Global override status := 0x0
[0649.821] I> MB1: MSS reconfig completed
[0649.830] I> Program carveout CARVEOUT_TEMP_MB2RF before image load
I> MB2 (version: 0.0.0.0-t234-54845784-a822a8cb)
I> t234-A01-0-Silicon (0x12347)
I> Boot-mode : SC7 Exit
I> Emulation:
I> Entry timestamp: 0x26bc32fd
I> Regular heap: [base:0x40040000, size:0x2000]
I> Task: ARI update carveout TZDRAM (0x40053cb0)
I> Task: Enable hot-plug capability (0x400609fc)
I> Task: PSC mailbox init (0x400588b4)
I> Task: Crypto init (0x40055a78)
I> Task: Enable GP-SE clock (0x40053db8)
I> Task: OEM SC7 context integrity check (0x400598bc)
I> Task: Restore FSI padctl config (0x400590ac)
I> Task: Program CBB PCIE AMAP regions (0x40059bc4)
I> Task: Load and authenticate registered FWs (0x4005b550)
I> Task: Load AUXP FWs (0x4006059c)
I> Skipping SCE FW load
I> Successfully register RCE FW load task with MB2 loader
I> Successfully register DCE FW load task with MB2 loader
I> Unpowergating APE
I> Unpowergate done
I> Successfully register APE FW load task with MB2 loader
I> Skipping FSI FW load
I> Successfully register XUSB FW load task with MB2 loader
I> rce: Authentication Finalize Done
I> Binary rce loaded successfully at 0x82d700000
I> dce: Authentication Finalize Done
I> Binary dce loaded successfully at 0x836000000
I> ape: Authentication Finalize Done
I> Binary ape loaded successfully at 0x82e400000
I> xusb: Authentication Finalize Done
I> Binary xusb loaded successfully at 0x82db00000
I> Task: Carveout setup (0x4005d8d0)
I> Program remaining OEM carveouts
I> Task: Enable FSITHERM (0x4005864c)
I> Task: Enable FSI VMON (0x40058148)
I> FSI VMON: FSI Vmon re-calibration and fine tuning done
I> Task: Validate FSI Therm readings (0x4005822c)
I> FSITHERM max temp = 32
I> FSITHERM min temp = 31
I> Task: Enable FSI SE clock (0x40058c54)
I> Enable FSI-SE clock and 10ms delay...
I> Task: Initialize SBSA UART CAR (0x40053d1c)
I> Task: Unpowergate APE (0x400594a4)
W> mb2_unpowergate_ape: skip! APE is in unpowergated state
I> Task: OEM firewalls (0x4005fb78)
I> OEM firewalls configured
I> Task: Powergate APE (0x40059618)
I> Powergating APE
I> Powergate done
I> Task: OEM firewall restore saved settings (0x4006001c)
I> Task: Deinit UART (0x40053bfc)
Reset reason 0x33
t234_adc_resume done
clk_mach_resume
clk_mach_resume done
跴pe exiting sc7
歊lk_mach_resume_late
?
  I> Task: Start secure NOR provision (0x400599dc)
I> Task: Load FSI keyblob (0x40058c9c)
I> Skipping FSI key blob copy
I> Task: Unhalt AUXPs (0x4006080c)
I> SCE unhalt skipped
I> Unhalting RCE
I> RCE unhalt successful
I> DCE unhalt successful
I> APE unhalt skipped
I> FSI unhalt skipped
I> Task: Load CPUBL (0x4嬂   649.970870] Camera-FW on t234-rce-safe started
TCU early console enabled.
?
  ?0597bc)
I> Task: Load TOS (0x400597e0)歊lk_mach_resume_late done
?
I> Task: Load and authenticate registered FWs (0x4005b550)
I> Task: Disable MSS perf stats (0x40060898)
I> Task: Program display sticky bits (0x40060814)
I> Task: SMMU external bypass disable (0x40058898)
I> Task: SMMU init (0x40058678)
I> Task: Program GICv3 registers (0x40060938)
I> Task: Audit firewall settings (0x4005fdd8)
I> MB2 finished榹TATUS_R[31:0]  = 0x1000100
STATUS_R[63:32] = 0x0
STATUS_R[95:64] = 0x100000
誏OTICE:  tegra_soc_pwr_domain_on_finish: exited SC7 successfully. Entering normal world.
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
I/TC: Secondary CPU 2 initializing
I/TC: Secondary CPU 2 switching to normal world boot
I/TC: Secondary CPU 3 initializing
I/TC: Secondary CPU 3 switching to normal world boot
I/TC: Secondary CPU 8 initializing
I/TC: Secondary CPU 8 switching to normal world boot
I/TC: Secondary CPU 9 initializing
I/TC: Secondary CPU 9 switching to normal world boot
I/TC: Secondary C嬂   650.183642] Camera-FW on t234-rce-safe ready SHA1=8676d22a (crt 2.938 ms, total boot 215.915 ms)
谽U 10 initializing
I/TC: Secondary CPU 10 switching to normal world boot
I/TC: Secondary CPU 11 initializing
? FW Boot DoneCPU 11 switching to normal world boot踜CE?

請問有任何建議嗎??

4
  1. enable more logs from pmc driver.
sudo tee /sys/kernel/debug/dynamic_debug/control <<< 'file drivers/soc/tegra/pmc.c +p'
  1. check logs after it has resumed

dmesg | grep “Resume caused”

example:
[ 1872.159407] tegra-pmc c360000.pmc: Resume caused by WAKE73, c2a0000.rtc
or
[ 67.637594] tegra-pmc c360000.pmc: Resume caused by WAKE29, Power

5

問題發生時 無法休眠結束回系統! 故無法取得kernel log …

6

Which jetpack version is this?

And I guess it is custom board?

7
  1. version JetPack_5.1.3_Jetson_Linux_35.5.0
  2. Yes
8

Hi,

想跟你先確認一下你們的cvb-> cvm power input 輸入電壓 最大電流是多少？

9

image

10

請問同樣一個module放到NV devkit上之後可以複製到問題嗎?

11

同樣一個module放到NV devkit 是正常的!! 可正確操作SC7 休眠 與 甦醒!

但在我司的板子 確有問題!
所以想知道 有合園因為導致這種狀況發生

12

Hi,

能否請你把pmc.c裡面的 “power” 從wake source先移除然後看一下這個問題會不會在你們板子發生嗎?

我不確定你用的release版本, 先放上rel-36的路徑
kernel/3rdparty/canonical/linux-jammy/kernel-source/drivers/soc/tegra/pmc.c

4419  static const struct tegra_wake_event tegra234_wake_events[] = {
4420  	TEGRA_WAKE_GPIO("power", 29, 1, TEGRA234_AON_GPIO(EE, 4)),

先看一下這個移除之後還有沒有發生問題. 這只是debug用的, 不是最終解

13

移除掉後 狀況一樣!!!

14

能請你加一些print確認你的patch有真的燒進去嗎?

15

image

image
image800×256 3.74 KB

16

能請你把 power & sd_wake 都移除掉嗎?

如果問題還在, 請你附上完整uart log. 不要只回一個 “問題還在”.

17

SDK版本

root@Jetson:/home/nvidia# cat /etc/nv_tegra_release
# R35 (release), REVISION: 5.0, GCID: 35550185, BOARD: t186ref, EABI: aarch64, DATE: Tue Feb 20 04:46:31 UTC 2024

[    4.258869] tegra_pmc_irq_alloc:pmu-209
[    4.264070] tegra_pmc_irq_alloc:rtc-10
[    4.268548] tegra_pmc_irq_alloc:sw_wake-179
[    4.272939] tegra_pmc_irq_alloc:pex_wake-0
[    4.277767] tegra_pmc_irq_alloc:usb3_port_0-167
[    4.282506] tegra_pmc_irq_alloc:usb3_port_1-167
[    4.287695] tegra_pmc_irq_alloc:usb3_port_2_3-167
[    4.292881] tegra_pmc_irq_alloc:usb2_port_0-167
[    4.298231] tegra_pmc_irq_alloc:usb2_port_1-167
[    4.303410] tegra_pmc_irq_alloc:usb2_port_2-167
[    4.308578] tegra_pmc_irq_alloc:usb2_port_3-167

202408221030_kernel.log (99.1 KB)

一樣會自動甦醒

?歊lk_mach_suspend_early
clk_mach_suspend_early done
suspended vdd_core @ 950000uV
跴pe entering sc7
歊lk_mach_suspend
clk_mach_suspend done
fmon_suspend done
adc_suspend done
WAKE_MASK[31:0]  = 0x1000000
WAKE_MASK[63:32] = 0x0
WAKE_MASK[95:64] = 0x1ff200
TIER0[31:0]      = 0x0
TIER0[63:32]     = 0x0
TIER0[95:64]     = 0x0
TIER1[31:0]      = 0x0
TIER1[63:32]     = 0x0
TIER1[95:64]     = 0x0
TIER2[31:0]      = 0x1000000
TIER2[63:32]     = 0x0
TIER2[95:64]     = 0xff200
[0343.566] I> MB1 (version: 1.4.0.1-t234-54845784-0db254a7)
[0343.572] I> t234-A01-0-Silicon (0x12347) Prod
[0343.578] I> Boot-mode : SC7 Exit
[0343.582] I> Entry timestamp: 0x00000000
[0343.586] I> last_boot_error: 0x0
[0343.590] I> BR-BCT: preprod_dev_sign: 0
[0343.595] I> rst_source: 0xc, rst_level: 0x3
[0343.600] I> Task: SE error check
[0343.604] I> Task: Enable SLCG
[0343.608] I> Task: CRC check
[0343.611] I> Skip FUSE records CRC check as records_integrity fuse is not burned

202408221030_sc7.log (8.7 KB)

18

ok. 能請你把wake event所有種類都移除掉 (tegra234_wake_events是空的) 然後再複製一次問題嗎

19

(tegra234_wake_events是空的)設定
休眠後不會自動甦醒

image
image1120×400 5.37 KB

現在在測試 個別單獨置能

20

感覺與PMU有關係
只單獨開PMU 會導致休眠後自動重啟

20241130_pmu_sc7.log (9.0 KB)

static const struct tegra_wake_event tegra234_wake_events[] = {	
	TEGRA_WAKE_IRQ("pmu", 24, 209),  	
};

只關PMU其他打開 休眠後不會自動重啟

static const struct tegra_wake_event tegra234_wake_events[] = {	
	TEGRA_WAKE_IRQ("usb2_port_0", 79, 167),
	TEGRA_WAKE_IRQ("usb2_port_1", 80, 167),
	TEGRA_WAKE_IRQ("usb2_port_2", 81, 167),
	TEGRA_WAKE_IRQ("usb2_port_3", 82, 167),	
	TEGRA_WAKE_IRQ("usb3_port_2_3", 78, 167),	
	TEGRA_WAKE_IRQ("usb3_port_0", 76, 167),
	TEGRA_WAKE_IRQ("usb3_port_1", 77, 167),
	TEGRA_WAKE_GPIO("sd_wake", 8, 0, TEGRA234_MAIN_GPIO(G, 7)),
	TEGRA_WAKE_GPIO("power", 29, 1, TEGRA234_AON_GPIO(EE, 4)),
	TEGRA_WAKE_IRQ("sw_wake", 83, 179),		
	TEGRA_WAKE_IRQ("rtc", 73, 10),	
	TEGRA_WAKE_GPIO("pex_wake", 1, 0, TEGRA234_MAIN_GPIO(L, 2)),	
	TEGRA_WAKE_GPIO("soc_gpio50", 48, 1, TEGRA234_AON_GPIO(BB, 2)),	//OK
	TEGRA_WAKE_GPIO("mgbe_wake", 56, 0, TEGRA234_MAIN_GPIO(Y, 3)),	//OK
	TEGRA_WAKE_GPIO("eqos_wake", 20, 0, TEGRA234_MAIN_GPIO(G, 4)),	//OK
};

20241130_other_sc7.log (1.3 KB)

21

Hi,

我們內部討論了一下
目前針對這個問題只能建議你們用示波器去觀察問題發生當時的電壓/電流，之後再檢查power-on sequence. 因為只有你們的CVB才會發生喚醒還不能resume成功的問題… 我們這邊能提供的幫助有限

22

感謝回覆!