Hi everyone,
We are facing an issue trying to enable PCIe Endpoint functionality on the AGX Orin DevKit (C5) using a internal non-spread spectrum clock (external clock disabled) based on PCIe endpoint clocking recommendation from the design guide.
The AGX Orin DevKit is connected to the x86 Host PC through a cable. The x86 PCIe clock has the spread spectrum disabled in the BIOS.
Here’s what I’ve done:
- Modified the BPMP file (tegra234-bpmp-3701-0000-3737-0000.dtb) to enable the internal clock (NVHS UPHY) and disable spread-spectrum.
- Modified the device tree (kernel_tegra234-p3701-0000-p3737-0000.dtb) to disable the external clock (“nvidia,enable-ext-refclk” removed) and enable SRNS (“nvidia,enable-srns” added).
- Followed the instructions in the NVIDIA documentation to enable PCIe Endpoint Mode: PCIe Endpoint Mode — Jetson Linux Developer Guide documentation
However, after starting the PCIe EP on the DevKit and powering on the PC, we are getting continuous error output and a crash.
root@jetson:/sys/kernel/config/pci_ep# cd /sys/kernel/config/pci_ep/
root@jetson:/sys/kernel/config/pci_ep# mkdir functions/pci_epf_nv_test/func1
root@jetson:/sys/kernel/config/pci_ep# echo 0x10de > functions/pci_epf_nv_test/func1/vendorid
root@jetson:/sys/kernel/config/pci_ep# echo 0x0001 > functions/pci_epf_nv_test/func1/deviceid
root@jetson:/sys/kernel/config/pci_ep# ln -s functions/pci_epf_nv_test/func1 controllers/141a0000.pcie_ep/
root@jetson:/sys/kernel/config/pci_ep# echo 1 > controllers/141a0000.pcie_ep/start
root@jetson:/sys/kernel/config/pci_ep#
root@jetson:/sys/kernel/config/pci_ep# sudo dmesg | tail
[ 18.443574] Adding 3501880k swap on /dev/zram7. Priority:5 extents:1 across:3501880k SS
[ 18.863172] fuse: init (API version 7.32)
[ 18.918783] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for aarch64 35.5.0 Release Build (buildbrain@mobile-u64-6519-d7000) Mon Feb 19 20:34:14 PST 2024
[ 19.007558] IPv6: ADDRCONF(NETDEV_CHANGE): l4tbr0: link becomes ready
[ 19.172997] NVRM rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0xffff:
[ 36.314615] pwm-tegra-tachometer 39c0000.tachometer: Tachometer Overflow is detected
[ 46.606975] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM phys: 0x131dfa000
[ 46.606993] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM IOVA: 0xffff0000
[ 46.607025] pci_epf_nv_test pci_epf_nv_test.0: BAR0 RAM virt: 0x0000000061ebd90c
root@jetson:/sys/kernel/config/pci_ep#
root@jetson:/sys/kernel/config/pci_ep#
------------------------POWER ON PCIe Root Complex-----------------------------------------
root@jetson:/sys/kernel/config/pci_ep# [ 53.268971] CPU:0, Error: cbb-fabric@0x13a00000, irq=22
[ 53.269136] **************************************
[ 53.269281] CPU:0, Error:cbb-fabric, Errmon:2
[ 53.269411] Error Code : TIMEOUT_ERR
[ 53.269524] Overflow : Multiple TIMEOUT_ERR
[ 53.269667]
[ 53.269721] Error Code : TIMEOUT_ERR
[ 53.269832] MASTER_ID : CCPLEX
[ 53.269931] Address : 0x3e90078
[ 53.270037] Cache : 0x1 -- Bufferable
[ 53.270154] Protection : 0x2 -- Unprivileged, Non-Secure, Data Access
[ 53.270348] Access_Type : Read
[ 53.270454] Access_ID : 0x17
[ 53.270455] Fabric : cbb-fabric
[ 53.270653] Slave_Id : 0x2e
[ 53.270734] Burst_length : 0x0
[ 53.270820] Burst_type : 0x1
[ 53.270916] Beat_size : 0x2
[ 53.271005] VQC : 0x0
[ 53.271147] GRPSEC : 0x7e
[ 53.271618] FALCONSEC : 0x0
[ 53.272079] AXI2APB_28_BLOCK_TMO_STATUS : 0x2
[ 53.272772] AXI2APB_28_BLOCK1_TMO : 0x10000
[ 53.273432] **************************************
[ 53.274196] WARNING: CPU: 0 PID: 197 at drivers/soc/tegra/cbb/tegra234-cbb.c:578 tegra234_cbb_isr+0x130/0x170
[ 53.280143] ---[ end trace 8dc5ff7311a56a17 ]---
[ 53.284698] CPU:0, Error: cbb-fabric@0x13a00000, irq=22
[ 53.289912] **************************************
[ 53.294898] CPU:0, Error:cbb-fabric, Errmon:2
[ 53.299190] Error Code : TIMEOUT_ERR
[ 53.302948] Overflow : Multiple TIMEOUT_ERR
[ 53.307678]
[ 53.309248] Error Code : TIMEOUT_ERR
[ 53.313009] MASTER_ID : CCPLEX
[ 53.316247] Address : 0x3e900c0
[ 53.319747] Cache : 0x1 -- Bufferable
[ 53.323949] Protection : 0x2 -- Unprivileged, Non-Secure, Data Access
[ 53.330686] Access_Type : Write
[ 53.334361] Access_ID : 0x0
[ 53.334362] Fabric : cbb-fabric
[ 53.341097] Slave_Id : 0x2e
[ 53.344247] Burst_length : 0x0
[ 53.347660] Burst_type : 0x1
[ 53.350984] Beat_size : 0x2
[ 53.354134] VQC : 0x0
[ 53.357022] GRPSEC : 0x7e
[ 53.360172] FALCONSEC : 0x0
[ 53.363149] AXI2APB_28_BLOCK_TMO_STATUS : 0x2
[ 53.367787] AXI2APB_28_BLOCK1_TMO : 0x10000
[ 53.372249] **************************************
[ 53.376988] WARNING: CPU: 0 PID: 197 at drivers/soc/tegra/cbb/tegra234-cbb.c:578 tegra234_cbb_isr+0x130/0x170
[ 53.386912] ---[ end trace 8dc5ff7311a56a18 ]---
Note, that endpoint functionality works as expected with the PCIe common clock configuration (when “nvidia,enable-ext-refclk” is set in the device tree) and the PC is able to properly enumerate and initialize the endpoint BARs.
Am I missing any steps to correctly enable PCIe endpoint function with a separate non-spread spectrum clock? Any help or suggestions would be greatly appreciated!
Both BPMB and kernel device tree files are provided for review.
kernel_tegra234-p3701-0000-p3737-0000.dtb.txt (429.1 KB)
tegra234-bpmp-3701-0000-3737-0000.dtb.txt (246.1 KB)
Thanks!