Pcie clk

wf1 · October 22, 2021, 9:24am

Hello everyone, I am applying Xavier+FPGA. I burned 32.5, and I used the driver on github. FPGA also downloaded the PCIE program. But my lspci can’t find the device.
The current hardware schematic diagram is as follows.

c491f63ec1413cd4b3ad1e2ac7eb3bb

Among them, PCIE_clkreq is pulled up. FPGA sets PCIE_CLKREQ to 0.
We grabbed from the FPGA side that PCIE_RST is always low, and reboot detects that it changes from low to high and then to low
I also found that pcie1 controller 4 in dts was not turned on. Now I turned it on, but the configuration may be incorrect. But there is still no clk.
this is dts ，/sys/firmware/fdt

pcie@14160000 {
		compatible = "nvidia,tegra194-pcie", "snps,dw-pcie";
		power-domains = <0x3 0x12>;
		reg = <0x0 0x14160000 0x0 0x20000 0x0 0x36000000 0x0 0x40000 0x0 0x36040000 0x0 0x40000>;
		reg-names = "appl", "config", "atu_dma";
		status = "okay";
		#address-cells = <0x3>;
		#size-cells = <0x2>;
		device_type = "pci";
		num-lanes = <0x2>;
		linux,pci-domain = <0x4>;
		clocks = <0x4 0xe0 0x4 0x143>;
		clock-names = "core_clk", "core_clk_m";
		resets = <0x5 0x7d 0x5 0x78>;
		reset-names = "core_apb_rst", "core_rst";
		interrupts = <0x0 0x33 0x4 0x0 0x34 0x4>;
		interrupt-names = "intr", "msi";
		iommus = <0x2 0x5a>;
		dma-coherent;
		#interrupt-cells = <0x1>;
		interrupt-map-mask = <0x0 0x0 0x0 0x0>;
		interrupt-map = <0x0 0x0 0x0 0x0 0x1 0x0 0x33 0x4>;
		nvidia,dvfs-tbl = <0xc28cb00 0xc28cb00 0xc28cb00 0x18519600 0xc28cb00 0xc28cb00 0x18519600 0x2faf0800 0xc28cb00 0x18519600 0x2faf0800 0x5f5e1000 0x0 0x0 0x0 0x0>;
		nvidia,max-speed = <0x2>;
		nvidia,disable-aspm-states = <0xf>;
		nvidia,controller-id = <0x3 0x4>;
		nvidia,disable-l1-cpm;
		nvidia,aux-clk-freq = <0x13>;
		nvidia,preset-init = <0x5>;
		nvidia,aspm-cmrt = <0x3c>;
		nvidia,aspm-pwr-on-t = <0x14>;
		nvidia,aspm-l0s-entrance-latency = <0x3>;
		bus-range = <0x0 0xff>;
		ranges = <0x81000000 0x0 0x36100000 0x0 0x36100000 0x0 0x100000 0x82000000 0x0 0x40000000 0x17 0x40000000 0x0 0xc0000000 0xc3000000 0x14 0x0 0x14 0x0 0x3 0x40000000>;
		nvidia,cfg-link-cap-l1sub = <0x1b0>;
		nvidia,cap-pl16g-status = <0x174>;
		nvidia,cap-pl16g-cap-off = <0x188>;
		nvidia,event-cntr-ctrl = <0x1c4>;
		nvidia,event-cntr-data = <0x1c8>;
		nvidia,margin-port-cap = <0x190>;
		nvidia,margin-lane-cntrl = <0x194>;
		nvidia,dl-feature-cap = <0x2f8>;
		nvidia,pex-wake = <0x13 0x5a 0x0>;
		vddio-pex-ctl-supply = <0xa>;
		nvidia,enable-power-down;
		phys = <0x14>;
		phy-names = "pcie-p2u-0";
		linux,phandle = <0xc5>;
		phandle = <0xc5>;
	};

What can I do to locate the problem now, thank you

linuxdev · October 22, 2021, 8:30pm

I cannot answer except for one common possibility: Is the FPGA fully booted prior to the Jetson booting? It is common for PCIe to “seem” to fail with an FPGA when it is really just that late PCI setup is disabled on the Jetson (for power saving), and that the FPGA just wasn’t ready to respond in time for early boot (if late setup were enabled, then in those cases the FPGA would again work…wiring could be correct, but timing of availability of FPGA could be wrong compared to timing of PCI bus enumeration).

wf1 · October 23, 2021, 8:01am

hi， it is reboot ，but it isnot work。

linuxdev · October 23, 2021, 2:45pm

Since it is not an issue of the FPGA booting too late the problem likely is something with device tree, but I am unable to say what that issue is.

vidyas · October 25, 2021, 5:10am

Hi,
Is this a customer IO board? or is it that you are using the Nvidia provided IO board itself?
If you are using an Nvidia-provided IO board since it has only M.2 Key-E and M.2 Key-M slots, how are you connecting your FPGA board to the system? any converters are being used?
From your summary, I observe that you are not seeing pcie@14160000 as not enabled in DT. That is strange because this is the C4 controller and owns M.2 Key-E slot in the default IO board. So, it must have been enabled by default. Did you de-compile the final DT that gets flashed onto the board?

wf1 · October 25, 2021, 5:20am

hi,
this is a customer IO board,connecting FPGA (xilinx xc7k160t) board,

wf1 · October 25, 2021, 5:30am

xilinx xc7k160t ， pcie is used xdma ip。 We grabbed from the FPGA side that PCIE_RST is always low, and reboot detects that it changes from low to high and then to low，PCIE1_CLK_P ，PCIE1_CLK_N is 0，lspci is nothing

wf1 · October 26, 2021, 10:02am

hi
pcie clk problem is solved，butI found a new problem，At the beginning，pcie is link ，then the later pcie is not link，Can you give me some advice？
dmesg | grep pci
[ 0.727835] iommu: Adding device 14160000.pcie to group 0
[ 0.728542] iommu: Adding device 141a0000.pcie to group 1
[ 1.983727] ehci-pci: EHCI PCI platform driver
[ 1.983774] ohci-pci: OHCI PCI platform driver
[ 2.419814] tegra-pcie-dw 14160000.pcie: Setting init speed to max speed
[ 2.420969] OF: PCI: host bridge /pcie@14160000 ranges:
[ 2.530849] tegra-pcie-dw 14160000.pcie: link is up
[ 2.531226] tegra-pcie-dw 14160000.pcie: PCI host bridge to bus 0004:00
[ 2.531235] pci_bus 0004:00: root bus resource [bus 00-ff]
[ 2.531244] pci_bus 0004:00: root bus resource [io 0x0000-0xfffff] (bus address [0x36100000-0x361fffff])
[ 2.531250] pci_bus 0004:00: root bus resource [mem 0x1740000000-0x17ffffffff] (bus address [0x40000000-0xffffffff])
[ 2.531254] pci_bus 0004:00: root bus resource [mem 0x1400000000-0x173fffffff pref]
[ 2.531284] pci 0004:00:00.0: [10de:1ad1] type 01 class 0x060400
[ 2.531455] pci 0004:00:00.0: PME# supported from D0 D3hot D3cold
[ 2.547738] pci 0004:00:00.0: PCI bridge to [bus 01-ff]
[ 2.547783] pci 0004:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 2.548003] pcieport 0004:00:00.0: Signaling PME through PCIe PME interrupt
[ 2.548011] pcie_pme 0004:00:00.0:pcie001: service driver pcie_pme loaded
[ 2.548110] aer 0004:00:00.0:pcie002: service driver aer loaded
[ 2.548314] pcie_pme 0004:00:00.0:pcie001: unloading service driver pcie_pme
[ 2.548374] aer 0004:00:00.0:pcie002: unloading service driver aer
[ 2.548476] pci_bus 0004:01: busn_res: [bus 01-ff] is released
[ 2.548588] pci_bus 0004:00: busn_res: [bus 00-ff] is released
[ 2.550182] tegra-pcie-dw 14160000.pcie: PCIe link is not up…!
[ 2.551358] tegra-pcie-dw 141a0000.pcie: Setting init speed to max speed
[ 2.552333] OF: PCI: host bridge /pcie@141a0000 ranges:
[ 2.662853] tegra-pcie-dw 141a0000.pcie: link is up
[ 2.663156] tegra-pcie-dw 141a0000.pcie: PCI host bridge to bus 0005:00
[ 2.663183] pci_bus 0005:00: root bus resource [bus 00-ff]
[ 2.663191] pci_bus 0005:00: root bus resource [io 0x100000-0x1fffff] (bus address [0x3a100000-0x3a1fffff])
[ 2.663197] pci_bus 0005:00: root bus resource [mem 0x1f40000000-0x1fffffffff] (bus address [0x40000000-0xffffffff])
[ 2.663201] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1f3fffffff pref]
[ 2.663226] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 2.663372] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[ 2.663960] pci 0005:01:00.0: [126f:2263] type 00 class 0x010802
[ 2.664075] pci 0005:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 2.675680] pci 0005:00:00.0: BAR 14: assigned [mem 0x1f40000000-0x1f400fffff]
[ 2.675689] pci 0005:01:00.0: BAR 0: assigned [mem 0x1f40000000-0x1f40003fff 64bit]
[ 2.675750] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[ 2.675758] pci 0005:00:00.0: bridge window [mem 0x1f40000000-0x1f400fffff]
[ 2.675775] pci 0005:00:00.0: Max Payload Size set to 128/ 256 (was 256), Max Read Rq 512
[ 2.675819] pci 0005:01:00.0: Max Payload Size set to 128/ 128 (was 128), Max Read Rq 512
[ 2.676098] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[ 2.676103] pci 0005:01:00.0: Signaling PME through PCIe PME interrupt
[ 2.676110] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[ 2.676264] aer 0005:00:00.0:pcie002: service driver aer loaded
[ 2.676667] nvme nvme0: pci function 0005:01:00.0

wf1 · October 26, 2021, 1:05pm

hi
pcie clk problem is solved，butI found a new problem，At the beginning，14160000.pcie is link ，then the later 14160000.pcie is not link，Can you give me some advice？
dmesg | grep pci
[ 0.727835] iommu: Adding device 14160000.pcie to group 0
[ 0.728542] iommu: Adding device 141a0000.pcie to group 1
[ 1.983727] ehci-pci: EHCI PCI platform driver
[ 1.983774] ohci-pci: OHCI PCI platform driver
[ 2.419814] tegra-pcie-dw 14160000.pcie: Setting init speed to max speed
[ 2.420969] OF: PCI: host bridge /pcie@14160000 ranges:
[ 2.530849] tegra-pcie-dw 14160000.pcie: link is up
[ 2.531226] tegra-pcie-dw 14160000.pcie: PCI host bridge to bus 0004:00
[ 2.531235] pci_bus 0004:00: root bus resource [bus 00-ff]
[ 2.531244] pci_bus 0004:00: root bus resource [io 0x0000-0xfffff] (bus address [0x36100000-0x361fffff])
[ 2.531250] pci_bus 0004:00: root bus resource [mem 0x1740000000-0x17ffffffff] (bus address [0x40000000-0xffffffff])
[ 2.531254] pci_bus 0004:00: root bus resource [mem 0x1400000000-0x173fffffff pref]
[ 2.531284] pci 0004:00:00.0: [10de:1ad1] type 01 class 0x060400
[ 2.531455] pci 0004:00:00.0: PME# supported from D0 D3hot D3cold
[ 2.547738] pci 0004:00:00.0: PCI bridge to [bus 01-ff]
[ 2.547783] pci 0004:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 2.548003] pcieport 0004:00:00.0: Signaling PME through PCIe PME interrupt
[ 2.548011] pcie_pme 0004:00:00.0:pcie001: service driver pcie_pme loaded
[ 2.548110] aer 0004:00:00.0:pcie002: service driver aer loaded
[ 2.548314] pcie_pme 0004:00:00.0:pcie001: unloading service driver pcie_pme
[ 2.548374] aer 0004:00:00.0:pcie002: unloading service driver aer
[ 2.548476] pci_bus 0004:01: busn_res: [bus 01-ff] is released
[ 2.548588] pci_bus 0004:00: busn_res: [bus 00-ff] is released
[ 2.550182] tegra-pcie-dw 14160000.pcie: PCIe link is not up…!
[ 2.551358] tegra-pcie-dw 141a0000.pcie: Setting init speed to max speed
[ 2.552333] OF: PCI: host bridge /pcie@141a0000 ranges:
[ 2.662853] tegra-pcie-dw 141a0000.pcie: link is up
[ 2.663156] tegra-pcie-dw 141a0000.pcie: PCI host bridge to bus 0005:00
[ 2.663183] pci_bus 0005:00: root bus resource [bus 00-ff]
[ 2.663191] pci_bus 0005:00: root bus resource [io 0x100000-0x1fffff] (bus address [0x3a100000-0x3a1fffff])
[ 2.663197] pci_bus 0005:00: root bus resource [mem 0x1f40000000-0x1fffffffff] (bus address [0x40000000-0xffffffff])
[ 2.663201] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1f3fffffff pref]
[ 2.663226] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 2.663372] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[ 2.663960] pci 0005:01:00.0: [126f:2263] type 00 class 0x010802
[ 2.664075] pci 0005:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[ 2.675680] pci 0005:00:00.0: BAR 14: assigned [mem 0x1f40000000-0x1f400fffff]
[ 2.675689] pci 0005:01:00.0: BAR 0: assigned [mem 0x1f40000000-0x1f40003fff 64bit]
[ 2.675750] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[ 2.675758] pci 0005:00:00.0: bridge window [mem 0x1f40000000-0x1f400fffff]
[ 2.675775] pci 0005:00:00.0: Max Payload Size set to 128/ 256 (was 256), Max Read Rq 512
[ 2.675819] pci 0005:01:00.0: Max Payload Size set to 128/ 128 (was 128), Max Read Rq 512
[ 2.676098] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[ 2.676103] pci 0005:01:00.0: Signaling PME through PCIe PME interrupt
[ 2.676110] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[ 2.676264] aer 0005:00:00.0:pcie002: service driver aer loaded
[ 2.676667] nvme nvme0: pci function 0005:01:00.0

d.hodge · November 3, 2021, 9:41am

This looks very similar to an issue that I’m having. How do you enable late PCI setup?

linuxdev · November 3, 2021, 4:22pm

I couldn’t tell you the specific setup for this. I’m sure it depends on which release. It is true that on a PC a hot detect is on by default (meaning software can run later, so don’t confuse this with hardware being safe to hot swap…some PCI devices need their own time to boot prior to the Jetson performing a query, and this is why PCI might need to be turned on and enabled to detect after a delay, e.g., some FPGAs need more time). I just don’t know the specific details, but you might want to start a new thread and ask after giving the exact hardware you are using.

Note that the reason PCI is normally off after initial boot is that it saves power.

d.hodge · November 4, 2021, 11:43am

Ok, thanks for the response. I’ll start a new thead

system · December 8, 2021, 12:57am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Jetson AGX Xavier Pcie(Root) detection of Device needs jetson reboot Jetson AGX Xavier pcie , fpga	25	2022	July 12, 2023
Use PCIe to communicate between two Xaviers: RP phy dosen't up Jetson AGX Xavier	17	2077	October 6, 2021
PCI Express card not being detected on boot occasionally Jetson AGX Xavier pcie , boot	11	2454	October 18, 2021
PCIE-HUB chip not being detected on boot occasionally Jetson Xavier NX pcie , boot , board-design	3	2712	October 18, 2021
Trouble getting any PCIE devices to show Jetson AGX Xavier pcie	13	1356	October 18, 2021
PCIE(C0) - is not working for Jetson AGX orin Jetson AGX Orin pcie	15	1055	October 10, 2023
Pcie offline Jetson AGX Orin pcie	13	1334	September 13, 2024
PCIE export error Jetson TX1	4	2737	July 5, 2018
TX1 <-> FPGA through PCIE Jetson TX1	42	15344	June 3, 2016
PCIe Status with FPGA card Jetson Xavier NX pcie , hw , kernel	4	1194	April 13, 2022

Pcie clk

Related topics