Confused about the number and configuration of PCIe lanes (Part 2)

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
#1

This is a continuation of the “Confused about the number of configuration of PCIe lanes” submitted in August of 2019.

I have the same problem with mapping of PCIe lanes. On the carrier board we designed for the Xavier AGX, we have PCIe lanes connected to UPHY8 and UPHY9. An FPGA is connected to these lanes, and I noticed that FPGA data is sending to the Xavier Module, but the Xavier isn’t sending any data back. Meaning that there isn’t any data on UPHY_TX8 and UPHY_TX9 lines. We do have another FPGA connected to UPHY2-5 lanes, and this one is working all the time. Are there any other settings that we need to do for UPHY8 and UPHY9 to work?

I also found that default PCIe configurations is as follow from NVIDIA Jetson Linux Developer Guide:
•C5: x8
•C0: x4
•C1, C3: x1

Does that mean that only these ports are usable? Also, is it true that we shouldn’t change the software configuration of these lanes?

Thanks,
Brittany

#3

UPHY lanes 8 & 9 are owned by PCIe C4 controller which is not enabled by default in AGX as there is no port to be owned by the C4 controller.
But, if it is a custom baseboard where there is a port and its lanes are connected to C4, then, it does work and for that, the C4 controller node in DT needs to be enabled.
The following change can be used for the same.
BTW, just enabling the C4 controller would only get the PCIe link up with the FPGA endpoint. You need to have a driver for the FPGA endpoint in the AGX host to configure and reserve memory and that’s when the data transfers from the FPGA endpoint start reflecting in AGXs system memory.

diff --git a/common/tegra194-p2888-0000-a00.dtsi b/common/tegra194-p2888-0000-a00.dtsi
index 2c5f7eb..4c01d89 100644
--- a/common/tegra194-p2888-0000-a00.dtsi
+++ b/common/tegra194-p2888-0000-a00.dtsi
@@ -133,6 +133,22 @@
phy-names = "pcie-p2u-0";
};
 
+ pcie@14160000 {
+ status = "okay";
+
+ vddio-pex-ctl-supply = <&p2888_spmic_sd3>;
+ nvidia,disable-aspm-states = <0xf>;
+ nvidia,enable-power-down;
+ nvidia,disable-clock-request;
+
+ nvidia,max-speed = <4>;
+
+ phys = <&p2u_8>,
+ <&p2u_9>;
+
+ phy-names = "pcie-p2u-0", "pcie-p2u-1";
+ };
+
pcie@141a0000 {
status = "disabled";
#4

Thanks for the quick response. I greatly appreciate it.
For our purpose, we currently are aiming to see if the PCIe lanes are present on Xavier, (mainly using lspci). Drivers for integrating the FPGA will be implemented later.

My next concern is stated in the previous post, saying:

Hi,

Yes, we don’t suggest to change the uPHY even though the software configuration is able to.

all the lanes are used and there are none available to get more than x8, x4 and x2 from those three controllers.

All the 12 lanes are shared by 3 usb ports and 4 pcie plus 1 UFS controller.
If you need help for more PCIe usecase, please send me a private message for your usecase.

If this is true, does that mean we shouldn’t change the device tree in order to enable C4 or change any of the uPHY lanes? What are the draw backs if we do decide to change the device tree?