Xavier PCIE Link Training problem

[thomas.gould]

We have custom carrier board with a 4 lane PCIe bus between an Xavier Jetson GPU and a Gen3 PCIe switch (PM8573). We have not been successful to meet our goal to have the link train to Gen3. When both the GPU and the PCIe switch are set to a maximum rate of Gen3, the link training reverts back to Gen1. If we set the maximum rate on the PCIe switch to Gen2, the link will train to Gen2. The link is very short and we have ruled out any signal integrity problems.

Questions:
What capabilities can we utilize on the GPU to trouble shoot our problem?
Can we force the GPU to send a compliance pattern at Gen3?
Are there registers other than listed in the TRM that would be useful?
Do you have any documentation you could provide on the Synopsys DesignWare PCIe Dual-Mode Controller?

Thank you

[kayccc]

Please refer to PCIe GEN3 PEX_CLK frequency - Jetson & Embedded Systems / Jetson AGX Xavier - NVIDIA Developer Forums to see if can help.

[k-hamada]

Please refer to PCIe GEN3 PEX_CLK frequency - Jetson & Embedded Systems / Jetson AGX Xavier - NVIDIA Developer Forums 2 to see if can help.

This linked topic may be about the PCIe clock, not the link train.

[WayneWWW]

Please check if your set the speed in device tree to only gen1.

[thomas.gould]

Thank you for your suggestion. We have the device tree speed set to Gen3. Attached are some LTSSM logs from the PCIe switch that might be helpful.

[thomas.gould]

and the last segment of the LTSSM log.

[k-hamada]

Hi,

Do you have a PCIe analyzer like

https://dl.cdn-anritsu.com/en-en/test-measurement/files/Brochures-Datasheets-Catalogs/datasheet/mp1900a-datasheet-e11400.pdf

?

The link is very short and we have ruled out any signal integrity problems.

The ‘link’ means your PCIe link, and ‘we have ruled out any signal integrity problems’ means that you’ve confirmed the signal integrity with your analyzer?

Kazu

[thomas.gould]

The PCB PCIe x4 lanes have been simulated to meet the requirements for Gen4. We are unable to connect a MP1900a type instrument due to the x4 lanes are directly from the GPU to the PCIe switch. We have been able to capture the Gen3 eye during link training and as you can see the eye looks good.

[WayneWWW]

Please check if this patch to your kernel helps or not.

diff --git a/drivers/pci/dwc/pcie-tegra.c b/drivers/pci/dwc/pcie-tegra.c
index 210c961..9aba346 100644
--- a/drivers/pci/dwc/pcie-tegra.c
+++ b/drivers/pci/dwc/pcie-tegra.c
@@ -3688,6 +3688,13 @@
 		writel(val, pci->dbi_base + EP_CFG_LINK_CAP);
 	}
 
+	if (pcie->init_speed >= 1 && pcie->init_speed <= 4) {
+		dw_pcie_read(pci->dbi_base + CFG_LINK_STATUS_CONTROL_2, 4, &val);
+		val &= ~CFG_LINK_STATUS_CONTROL_2_TARGET_LS_MASK;
+		val |= pcie->init_speed;
+		dw_pcie_write(pci->dbi_base + CFG_LINK_STATUS_CONTROL_2, 4, val);
+	}
+
 	writew(PCI_CLASS_MEMORY_OTHER,
 	       pci->dbi_base + PCI_CLASS_DEVICE);

[k-hamada]

Hi, Mr. thomas.gould

Thank you for your eye diagram, but you have also good response from the PCIe switch PM8573 ?

Kazu

[thomas.gould]

Thank you Wayne for the patch. Unfortunately that patch did not solve our problem but thanks anyway.

[k-hamada]

Hi, Mr. thomas.gould

I strongly recommend you to check the signal integrity once more especially in the return path from PCIe switch PM8573 to Xavier. And also check the power line of PM8573 if it has enough low impedance.