Occasionally `14160000.pcie: Phy link never came up`

Hi guys, I met a problem that we cannot enumerate a 10G network card on 14160000.pcie. We use custom board, and the recurrence probability is low(1/50) in some carrier board and that may be higher in other board(3/5). Here is the full logs, containing four cases: the cold/hot boot + success/failure of enumerating the PCIe:

AQC113_cannot_found_logs.tar.gz (88.5 KB)

I try to extend the waiting time of PCIe by the following patch:

diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index 5563979310ba..fe8d790db305 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -557,6 +557,7 @@ int dw_pcie_wait_for_link(struct dw_pcie *pci)
                        return 0;
                usleep_range(LINK_WAIT_USLEEP_MIN, LINK_WAIT_USLEEP_MAX);
+               dev_info(pci->dev, "ben: wait loop (%d/%d)\n", retries+1, LINK_WAIT_MAX_RETRIES);

        dev_info(pci->dev, "Phy link never came up\n");
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 76c57d4fa714..fd4c78dab204 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -25,7 +25,7 @@
 #define DW_PCIE_VER_562A               0x3536322A

 /* Parameters for the waiting for link up routine */
-#define LINK_WAIT_MAX_RETRIES          10
+#define LINK_WAIT_MAX_RETRIES          100
 #define LINK_WAIT_USLEEP_MIN           90000
 #define LINK_WAIT_USLEEP_MAX           100000

Sometimes the enumeration success at the first try as before. While sometimes it needs about 30 tries, sometimes it needs about 60 tries and other times it will be failed even after100 tries. Any advice or suspicion do you have?

Does previous problem get resolved or not?

No, as this problem stuck the test. Tomorrow, I’ll skip it and continue testing.

Could you use rel-35 to test this first to make sure rel-35 is fine?

Rel-36 is still a developer preview which is not proper for bring up.

The problem is also occured with rel-35.3.1 as we tested before.

By the way, we’ve test such case yesterday:

I use a usb stick containing a rootfs and then chroot to that environment. Sometimes the pcie could be enumerate by reloading pcie_tegra194.


What is the exact issue you are talking about here? Why is usb stick + chroot needed? It sounds totally not related to PCIe.

Sorry for confusing. The initial purpose is that we try to reload the driver to see if this help. As the external disk is also on the PCIe bus, we have to use a usb stick to store the rootfs.

Let’s clarify the whole situation again

  1. What is the exact pcie device that will lead the this error? You said 10G network card on 14160000.pcie would casue problem in the beginning.

  2. Now you say the external disk also on the PCIe bus.

Which one is causing the error here? Both devices could cause boot hang?

The first one, only 10G network card cause the problem.

The reason I mentioned external disk is that if I unload the pcie_tegra194, the external disk will off. The whole system is stored in that, so I cannot “reload the pcie driver” without helping of usb disk. The “external disk” I mean here is the nvme disk on pcie bus.

Sorry again for confusing. Please ignore the usb stick or disk, the point is that sometimes reloading the pcie driver can enumerate the 10G network card.

Is it possible to test same 10G card on NVIDIA devkit instead of your board?

Oh. Sorry. I remember that now. You are the guy who has no devkit.

If rel-35 has no stuck problem, then why not just use bind and unbind to test?

Then you don’t need to use usb stick at all.

OK, I’ll try that. Thanks for your such patient reply.

new phenomenon? the device was refound after the print Phy link never came up

[    5.928579] tegra194-pcie 14160000.pcie: Adding to iommu group 7
[    5.940982] tegra194-pcie 14160000.pcie: Using GICv2m MSI allocator
[    7.631488] tegra194-pcie 14160000.pcie: Using GICv2m MSI allocator
[    7.639410] tegra194-pcie 14160000.pcie: host bridge /pcie@14160000 ranges:
[    7.646580] tegra194-pcie 14160000.pcie:       IO 0x0036100000..0x00361fffff -> 0x0036100000
[    7.655263] tegra194-pcie 14160000.pcie:      MEM 0x2428000000..0x242fffffff -> 0x0040000000
[    7.663937] tegra194-pcie 14160000.pcie:      MEM 0x2140000000..0x2427ffffff -> 0x2140000000
[    8.779725] tegra194-pcie 14160000.pcie: Phy link never came up
[    8.785868] tegra194-pcie 14160000.pcie: PCI host bridge to bus 0004:00
[    8.792677] pci_bus 0004:00: root bus resource [bus 00-ff]
[    8.798316] pci_bus 0004:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x36100000-0x361fffff])
[    8.808410] pci_bus 0004:00: root bus resource [mem 0x2428000000-0x242fffffff] (bus address [0x40000000-0x47ffffff])
[    8.819232] pci_bus 0004:00: root bus resource [mem 0x2140000000-0x2427ffffff pref]
[    8.827148] pci 0004:00:00.0: [10de:229c] type 01 class 0x060400
[    8.833455] pci 0004:00:00.0: PME# supported from D0 D3hot
[    8.845912] pci 0004:00:00.0: PCI bridge to [bus 01-ff]
[    8.851303] pci 0004:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.860210] pcieport 0004:00:00.0: Adding to iommu group 7
[    8.866045] pcieport 0004:00:00.0: PME: Signaling with IRQ 57
[    8.872453] pcieport 0004:00:00.0: AER: enabled with IRQ 57
[    8.875493] pcieport 0004:00:00.0: AER: Corrected error received: 0004:00:00.0
[    8.885631] pcieport 0004:00:00.0: AER: can't find device of ID0000
[    8.892863] pci_bus 0004:01: busn_res: [bus 01-ff] is released
[    8.898975] pci 0004:00:00.0: Removing from iommu group 7
[    8.904539] pci_bus 0004:00: busn_res: [bus 00-ff] is released

Here is the full_log:
the_ng.log (94.7 KB)

Besides, I compare the kernel output between two boots, before the PCIe never link up, there is such difference( left is the ok boot, while the right is the ng boot:

It seems that the ok boot contains the following log while ng boot not:

dce: dce_admin_ipc_handle_signal:90   Spurious signal on channel: [1]. Ignored...

And here is the two logs:
dmesg_ok.log (66.7 KB)
dmesg_ng.log (70.0 KB)

I see, the DCE means display controller engine. There may not has much relation.

Hi WayneWWW. After rechecking the release note of AQC113C, we may found out the reason of the problem. As the release note said, we need to switch to new method:

original method:

  1. No Link
  2. Train to Gen 1
  3. Immediately speed change and train to Gen 3 (usually fails at this location)
  4. Immediately speed change and train to Gen 4

new method:

  1. No Link
  2. Set host Train to Gen 1
  3. Confirm Gen 1 linked up
  4. Proceed to link to Gen 3
  5. Proceed to speed change and link to Gen 4

We’ve check the device tree documents, and found out that there is a property called nvidia,init-link-speed which may do this. But we does not find out where it is been used. We also tried with the config set to <0x1>, but it does not take effect.


  1. If the tegra PCIe using the new method?
  2. If the answer to Q1 is no, where is the implement of nvidia,init-link-speed if it is been implement?
  3. For this circumstance, Could you help us solving it? If all done in kernel is complicated, may we set the speed with gen-1 during boot, and then set the speed to gen-4 from user space through sysfs?

Could you try to set nvidia,init-speed = 0x1?

With nvidia,init-speed = 0x1; I failed to build the dtb. So I change it to nvidia,init-speed = <0x1>;. But it does not help with enumerating the pcie device.

Will max-link-speed = <1> give you the enumeration?

Yes, with about 700 reboots, all the enumeration works.