I am connecting an FPGA based PCIe Device to Jetson AGX Xavier PCIe slot.
I am facing an issue with device detection.
Device will get detected properly If I comment “nvidia,enable-power-down;” in device-tree file “tegra194-p2888-0000-a00.dtsi”. If it is not-commented I need to reboot my Jetson 2-3 times to see the device got detected in lspci-v.
Can you please help with thhis.
Best Regards,
Prasanna
Hi @prasanna, I think you will receive better support if we put this post in the Jetson AGX category.
I hope you don’t mind me moving it for you.
Thanks!
Hi,
Could you check if you can get the device gets detected if you directly rescan the pcie bus instead of doing reboot?
If you need to disable “nvidia,enable-power-down” to make pcie device always gets detected, it may indicate there are some other power control that would affect your device. For example, maybe need to wait for specific GPIO to be ON so that your device can work. Sometimes PCIe controller is enabled earlier than that GPIO and your device cannot get detected.
Hello Wayne,
Thanks for the response.
I have tried rescan. device didn’t get detected.
Here are few of the exercises that I followed :
To provide some more time to my device to get powered on completely. I delayed jetson booting by stopping the auto-boot in u-boot by using a [J501] UART console. During that time also device not detected.
I have gone through “/sources/kernel/nvidia/drivers/pci/dwc/pcie-tegra.c”.
Here are few observations. in “tegra_pcie_dw_host_init”, In working scenario CFG_LINK_STATUS_CONTROL status will active and the link will be up. where-as in failed case “link is down”. So I tried increasing the retry count for checking the link status. That also didn’t help.
in “tegra_pcie_config_rp” function before checking for the “tegra_pcie_dw_link_up(&pcie->pci);” I provided some delay. That also didn’t help.
Here is the snippet.
pm_runtime_enable(pcie->dev);
ret = pm_runtime_get_sync(pcie->dev);
if (ret < 0) {
dev_err(pcie->dev, “failed to enable pcie dev”);
pm_runtime_disable(pcie->dev);
return ret;
}
++ msleep(4000); /Custom Added/
pcie->link_state = tegra_pcie_dw_link_up(&pcie->pci);
if (!pcie->link_state && pcie->power_down_en) {
ret = -ENOMEDIUM;
goto fail_host_init;
}
I have attached working and non-working logs. in working case x8 will get detected “141a0000.pcie:”.
Please let me know any additional input needed for you.
Is there any way. That I can re-initiate the pcie enumeration sequence manually. So that I can try and share the outcome.
Best Regards,
Prasanna
Xavier does not use uboot. So not sure about what auto-boot you are telling. Or you are talking about cboot?
You didn’t attach any log here…
Please try to build the pcie driver into module. Load/unload it after boot into kernel manually and see if device would be up in such way.
Hello Wayne,
Sorry missed the attachments. Please find the attachments.
Non_Working.txt (4.3 KB)
Working.txt (4.2 KB)
I tried to build the pcie-driver as module. I used pcie-tegra.c and pcie-designware. But it asked for some dependencies. Could you please help with which are the files to be used and build the pcie driver. would be easy. So that I will try it.
Thansk,
Prasanna
Hi,
For any log, please just share full one. You don’t need to parse anything (e.g. grep pcie).
Also, please share the result of lspci -vvv in both working and non working case.
Hi,
Please find the attached logs for Working and Non-Working.
Non_working.txt (72.1 KB)
non_working_lspci (1.8 KB)
After I reboot My Jetson 4 times. Both my X8 and X4 devices will get detected.
working.txt (73.3 KB)
working_lspci (4.7 KB)
Thanks,
Prasanna
Hi,
Have you tried rebind instead of rescan?
Something like below. Change 14160000 to your specific pcie controller.
cd /sys/bus/platform/drivers/tegra194-pcie
echo 14160000.pcie > unbind
echo 14160000.pcie > bind
Hi.
I tried as you suggested unbind and bind. Below is the outcome of dmesg. Still link is not up.
root@jetson6:/home/jetson6# cd /sys/bus/platform/drivers/tegra-pcie-dw/
root@jetson6:/sys/bus/platform/drivers/tegra-pcie-dw# ls -l
total 0
lrwxrwxrwx 1 root root 0 Feb 9 14:55 14100000.pcie → …/…/…/…/devices/14100000.pcie
lrwxrwxrwx 1 root root 0 Feb 9 14:55 14140000.pcie → …/…/…/…/devices/14140000.pcie
lrwxrwxrwx 1 root root 0 Feb 9 14:55 14180000.pcie → …/…/…/…/devices/14180000.pcie
lrwxrwxrwx 1 root root 0 Feb 9 14:55 141a0000.pcie → …/…/…/…/devices/141a0000.pcie
–w------- 1 root root 4096 Feb 9 14:55 bind
–w------- 1 root root 4096 Feb 9 14:44 uevent
–w------- 1 root root 4096 Feb 9 14:55 unbind
root@jetson6:/sys/bus/platform/drivers/tegra-pcie-dw#
Unbind - output of dmesg
371.420695] OF: PCI: host bridge /pcie@14180000 ranges:
[ 371.420742] OF: PCI: IO 0x38100000…0x381fffff → 0x38100000
[ 371.420754] OF: PCI: MEM 0x1b40000000…0x1bffffffff → 0x40000000
[ 371.420762] OF: PCI: MEM 0x1800000000…0x1b3fffffff → 0x1800000000
[ 371.933147] tegra-pcie-dw 14180000.pcie: link is down
[ 371.933680] tegra-pcie-dw 14180000.pcie: PCI host bridge to bus 0000:00
[ 371.933693] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 371.933713] pci_bus 0000:00: root bus resource [io 0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[ 371.933722] pci_bus 0000:00: root bus resource [mem 0x1b40000000-0x1bffffffff] (bus address [0x40000000-0xffffffff])
[ 371.933731] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1b3fffffff pref]
[ 371.933776] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 371.933967] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[ 371.934598] iommu: Adding device 0000:00:00.0 to group 64
[ 371.935131] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 371.935164] pci 0000:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 371.935854] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[ 371.935872] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[ 371.940823] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 371.941061] tegra-asoc: sound: snd_soc_register_card failed (-517)
[ 371.943791] aer 0000:00:00.0:pcie002: service driver aer loaded
[ 371.947966] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 371.948248] tegra-asoc: sound: snd_soc_register_card failed (-517)
[ 371.950562] pcie_pme 0000:00:00.0:pcie001: unloading service driver pcie_pme
[ 371.950823] aer 0000:00:00.0:pcie002: unloading service driver aer
[ 371.951089] pci_bus 0000:01: busn_res: [bus 01-ff] is released
[ 371.951314] iommu: Removing device 0000:00:00.0 from group 64
[ 371.951383] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[ 371.959401] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 371.959629] tegra-asoc: sound: snd_soc_register_card failed (-517)
[ 371.961454] tegra-pcie-dw 14180000.pcie: PCIe link is not up…!
Bind output of dmesg
[ 409.556301] tegra-pcie-dw 14180000.pcie: Setting init speed to max speed
[ 409.557223] tegra-pcie-dw 14180000.pcie: Unbalanced pm_runtime_enable!
[ 409.557878] OF: PCI: host bridge /pcie@14180000 ranges:
[ 409.557905] OF: PCI: IO 0x38100000…0x381fffff → 0x38100000
[ 409.557916] OF: PCI: MEM 0x1b40000000…0x1bffffffff → 0x40000000
[ 409.557922] OF: PCI: MEM 0x1800000000…0x1b3fffffff → 0x1800000000
[ 410.072692] tegra-pcie-dw 14180000.pcie: link is down
[ 410.073109] tegra-pcie-dw 14180000.pcie: PCI host bridge to bus 0000:00
[ 410.073124] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 410.073148] pci_bus 0000:00: root bus resource [io 0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[ 410.073157] pci_bus 0000:00: root bus resource [mem 0x1b40000000-0x1bffffffff] (bus address [0x40000000-0xffffffff])
[ 410.073168] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1b3fffffff pref]
[ 410.073210] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 410.073402] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[ 410.074051] iommu: Adding device 0000:00:00.0 to group 64
[ 410.074442] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 410.074507] pci 0000:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 410.074878] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[ 410.074891] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[ 410.075057] aer 0000:00:00.0:pcie002: service driver aer loaded
[ 410.075315] pcie_pme 0000:00:00.0:pcie001: unloading service driver pcie_pme
[ 410.075433] aer 0000:00:00.0:pcie002: unloading service driver aer
[ 410.075620] pci_bus 0000:01: busn_res: [bus 01-ff] is released
[ 410.081426] iommu: Removing device 0000:00:00.0 from group 64
[ 410.081608] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[ 410.083210] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 410.083453] tegra-asoc: sound: snd_soc_register_card failed (-517)
[ 410.084580] tegra-pcie-dw 14180000.pcie: PCIe link is not up…!
[ 410.090217] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 410.090441] tegra-asoc: sound: snd_soc_register_card failed (-517)
Do I need to perform multiple times ?
Hi,
Do both C0 and C5 controller all get the same result after bind/unbind? Your log only shows C0.
I tried only for C0. I will try for both and share.
If I need to build pcie driver as a module. can provide details which section of files I need to use and load the modules explicitly and try.
Thanks,
Prasanna
Here is the output for C5 also. Out put is remain same
root@jetson15:/home/jetson15# dmesg --follow
Unbind
[ 491.458492] OF: PCI: host bridge /pcie@141a0000 ranges:
[ 491.458548] OF: PCI: IO 0x3a100000…0x3a1fffff → 0x3a100000
[ 491.458562] OF: PCI: MEM 0x1f40000000…0x1fffffffff → 0x40000000
[ 491.458626] OF: PCI: MEM 0x1c00000000…0x1f3fffffff → 0x1c00000000
[ 491.978195] tegra-pcie-dw 141a0000.pcie: link is down
[ 491.978650] tegra-pcie-dw 141a0000.pcie: PCI host bridge to bus 0005:00
[ 491.978667] pci_bus 0005:00: root bus resource [bus 00-ff]
[ 491.978721] pci_bus 0005:00: root bus resource [io 0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[ 491.978731] pci_bus 0005:00: root bus resource [mem 0x1f40000000-0x1fffffffff] (bus address [0x40000000-0xffffffff])
[ 491.978741] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1f3fffffff pref]
[ 491.978792] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 491.978986] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[ 491.979657] iommu: Adding device 0005:00:00.0 to group 64
[ 491.980330] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[ 491.980368] pci 0005:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 491.980912] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[ 491.980926] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[ 491.981283] aer 0005:00:00.0:pcie002: service driver aer loaded
[ 491.981539] pcie_pme 0005:00:00.0:pcie001: unloading service driver pcie_pme
[ 491.981707] aer 0005:00:00.0:pcie002: unloading service driver aer
[ 491.987924] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 491.988198] tegra-asoc: sound: snd_soc_register_card failed (-517)
[ 491.994681] pci_bus 0005:01: busn_res: [bus 01-ff] is released
[ 491.994970] iommu: Removing device 0005:00:00.0 from group 64
[ 491.995082] pci_bus 0005:00: busn_res: [bus 00-ff] is released
[ 491.998857] tegra-pcie-dw 141a0000.pcie: PCIe link is not up…!
Bind
[ 503.002495] tegra-pcie-dw 141a0000.pcie: Setting init speed to max speed
[ 503.004355] tegra-pcie-dw 141a0000.pcie: Unbalanced pm_runtime_enable!
[ 503.005194] OF: PCI: host bridge /pcie@141a0000 ranges:
[ 503.005240] OF: PCI: IO 0x3a100000…0x3a1fffff → 0x3a100000
[ 503.005254] OF: PCI: MEM 0x1f40000000…0x1fffffffff → 0x40000000
[ 503.005262] OF: PCI: MEM 0x1c00000000…0x1f3fffffff → 0x1c00000000
[ 503.522064] tegra-pcie-dw 141a0000.pcie: link is down
[ 503.522503] tegra-pcie-dw 141a0000.pcie: PCI host bridge to bus 0005:00
[ 503.522524] pci_bus 0005:00: root bus resource [bus 00-ff]
[ 503.522548] pci_bus 0005:00: root bus resource [io 0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[ 503.522560] pci_bus 0005:00: root bus resource [mem 0x1f40000000-0x1fffffffff] (bus address [0x40000000-0xffffffff])
[ 503.522568] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1f3fffffff pref]
[ 503.522617] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[ 503.522810] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[ 503.523339] iommu: Adding device 0005:00:00.0 to group 64
[ 503.523822] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[ 503.523856] pci 0005:00:00.0: Max Payload Size set to 256/ 256 (was 256), Max Read Rq 512
[ 503.524332] pcieport 0005:00:00.0: Signaling PME through PCIe PME interrupt
[ 503.524344] pcie_pme 0005:00:00.0:pcie001: service driver pcie_pme loaded
[ 503.530203] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 503.530458] tegra-asoc: sound: snd_soc_register_card failed (-517)
[ 503.535707] aer 0005:00:00.0:pcie002: service driver aer loaded
[ 503.539876] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 503.540170] tegra-asoc: sound: snd_soc_register_card failed (-517)
[ 503.542176] pcie_pme 0005:00:00.0:pcie001: unloading service driver pcie_pme
[ 503.542346] aer 0005:00:00.0:pcie002: unloading service driver aer
[ 503.542633] pci_bus 0005:01: busn_res: [bus 01-ff] is released
[ 503.542808] iommu: Removing device 0005:00:00.0 from group 64
[ 503.542899] pci_bus 0005:00: busn_res: [bus 00-ff] is released
[ 503.545728] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 503.545993] tegra-asoc: sound: snd_soc_register_card failed (-517)
[ 503.546050] tegra-pcie-dw 141a0000.pcie: PCIe link is not up…!
[ 503.553844] tegra-asoc: sound: ASoC: CODEC DAI rt5659-aif1 not registered
[ 503.554117] tegra-asoc: sound: snd_soc_register_card failed (-517)
Hi,
One question here. Your log indicates the kernel version seems a little bit old. Probably rel-32.5 or rel-32.4.4 I guess, could you also upgrade to rel-32.7.3?
Hi,
Here is my kernel details.
and the kernel version is “Linux jetson 4.9.140-tegra #4 SMP PREEMPT Tue Mar 23 11:17:00 IST 2021 aarch64 aarch64 aarch64 GNU/Linux”.
tegra-l4t-r32.4.3
I can upgrade and check. Do you have any idea why this behavior is observed with older kernel?
To build tegra-pcie-dw module, Could you help with the dependent files to be used. So that I can build and test that as well.
So will share the both the outcome.
Thanks,
Prasanna
To build tegra-pcie-dw module, Could you help with the dependent files to be used. So that I can build and test that as well.
This could be pending. Does not really matter at this moment.
Hi,
We have some internal check about this issue. Please just keep removing nvidia,enable-power-down and as a solution.
Your PCIe EP seems a old one which needs DLFE disable.
Hello Wayne,
Actually in the pcie-tegra driver module it will also try by disabling the DLFE as well.
dw_pcie_read(pci->dbi_base + pcie->dl_feature_cap, 4, &val);
val &= ~DL_FEATURE_EXCHANGE_EN;
dw_pcie_write(pci->dbi_base + pcie->dl_feature_cap, 4, val);
Even I have done this experiment by keeping DLFE Disabled by default in “tegra_pcie_dw_host_init” that also gave the same result.
Now related to disabling “nvidia,enable-power-down” in device-tree. I have below queries. Please let us know your input. So that till it get checked and provided the solution from nVidia I can use by keeping “nvidia,enable-power-down” in disable.
Q1 : If no PCIe device is connected to the Jetson, in that case since power-down is disabled. a continuous PCIe clock will be present. Since on X8 interface there is termination will it impact anything to the Jetson ?
Q2: Since my PCIe Device is self powered. In any scenario. If Device is OFF and Jetson is ON. and Jetson provides the PCIe Clock continuously will there be any impact on the PCIe EP device ??
Let us know your thoughts on above.
Best Regards,
Prasanna
Hi Wayne,
Any update on my previous questions ??
If I use Jetpack 5.1 My PCIe device (EP) are getting for the first time itself.
Could you please provide the details/differences in Jetpack 5.1 related to PCIe drivers.
Best Regards,
Prasanna
Hello Wayne,
Do you have any update on this ?
Regards,
Prasanna