TX2 Custom carrier board cannot recognize all PCIe devices

Hi,
I recently customized a TX2 carrier board and modified the device tree to configure 5: 1xUSB3.0+2x1 PCIe+1x2 PCIe. I have checked ODMDATA using the busybox command, so the current device tree should be correct. The strange thing is that I can only detect one of the PCIe devices, and the other two cannot be detected. I want to ask:

  1. Why is this?
  2. Which equipment can I work on?

Here are some information I can provide:

dmesg.txt (60.0 KB)
lspci.txt (1.3 KB)


Could you also post your dts file?

Hi,WayneWWW
dts.zip (8.1 KB)
//JetPack 4.3

Hi,WayneWWW
I want to correct my second question. Exactly, I want to know which PCIe device is working.PEX0,PEX1orPEX2ļ¼Ÿ

Hi,

  1. For your dts file, please directly convert the dtb file back to dts with dtc tool and paste it here.

  2. And for your question 2, I tend to not answer this question. You should be able to find out which pice device is working by either hardware checking of software check by yourself.

Thank you for your reply.

  1. This is my dts file:

tegra186.zip (59.1 KB)

  1. According to dmesg and Design Guide, I think
    the working pcie device is pcie#1. Module Pin Names:PEX2. Tegra Lanes:Lane 3

Hi,
Is there something wrong with my dts file?

Please also disable non-necessary usb devices.

			usb3-0 {
					status = "okay";
					#phy-cells = <0x0>;
					nvidia,function = "xusb";
					linux,phandle = <0x107>;
					phandle = <0x107>;
				};
				usb3-1 {
					status = "okay";
					#phy-cells = <0x0>;
					nvidia,function = "xusb";
					linux,phandle = <0xad>;
					phandle = <0xad>;
				};
				usb3-2 {
					status = "okay";
					#phy-cells = <0x0>;
					nvidia,function = "xusb";
				};

Hi,

Thank you for your guidance. With your help, I can make two PCIe devices work, but one still doesnā€™t work.

But I canā€™t upload my file anymore, it will return such an error after uploading any file:

undefined method `hostnameā€™ for nil:NilClass

It is still not possible to upload files.

However, I found that the two devices that can work now are exactly the two devices that did not work before, exactly,

Only PCIe 1# can work before modifying the dts,
only PCIe 1# canā€™t work after modifying the dts.

The following is the output of command ā€˜dmesg | grep pciā€™

[ 0.513981] GPIO line 459 (pcie0_lane2_mux) hogged as output/low
[ 0.524213] iommu: Adding device 10003000.pcie-controller to group 49
[ 0.524287] arm-smmu: forcing sodev map for 10003000.pcie-controller
[ 0.980166] tegra-pcie 10003000.pcie-controller: 2x1, 1x1, 1x1 configuration
[ 0.981323] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
[ 0.981769] tegra-pcie 10003000.pcie-controller: probing port 0, using 2 lanes
[ 0.985253] tegra-pcie 10003000.pcie-controller: probing port 1, using 1 lanes
[ 0.987526] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
[ 1.549083] tegra-pcie 10003000.pcie-controller: link 1 down, retrying
[ 1.957167] tegra-pcie 10003000.pcie-controller: link 1 down, retrying
[ 2.365175] tegra-pcie 10003000.pcie-controller: link 1 down, retrying
[ 2.367231] tegra-pcie 10003000.pcie-controller: link 1 down, ignoring
[ 2.476590] tegra-pcie 10003000.pcie-controller: PCI host bridge to bus 0000:00
[ 2.476602] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 2.476608] pci_bus 0000:00: root bus resource [mem 0x40100000-0x47ffffff]
[ 2.476614] pci_bus 0000:00: root bus resource [mem 0x48000000-0x7fffffff pref]
[ 2.476621] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.476666] pci 0000:00:01.0: [10de:10e5] type 01 class 0x060400
[ 2.476805] pci 0000:00:01.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2.477287] pci 0000:00:03.0: [10de:10e6] type 01 class 0x060400
[ 2.477411] pci 0000:00:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[ 2.477752] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2.477769] pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 2.478019] pci 0000:01:00.0: [1912:0015] type 00 class 0x0c0330
[ 2.478325] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[ 2.478658] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 2.488196] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 2.488436] pci 0000:02:00.0: [1912:0015] type 00 class 0x0c0330
[ 2.488503] pci 0000:02:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[ 2.488812] pci 0000:02:00.0: PME# supported from D0 D3hot
[ 2.504169] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[ 2.504251] pci 0000:00:01.0: BAR 14: assigned [mem 0x40100000-0x401fffff]
[ 2.504257] pci 0000:00:03.0: BAR 14: assigned [mem 0x40200000-0x402fffff]
[ 2.504268] pci 0000:01:00.0: BAR 0: assigned [mem 0x40100000-0x40101fff 64bit]
[ 2.504304] pci 0000:00:01.0: PCI bridge to [bus 01]
[ 2.504316] pci 0000:00:01.0: bridge window [mem 0x40100000-0x401fffff]
[ 2.504334] pci 0000:02:00.0: BAR 0: assigned [mem 0x40200000-0x40201fff 64bit]
[ 2.504367] pci 0000:00:03.0: PCI bridge to [bus 02]
[ 2.504378] pci 0000:00:03.0: bridge window [mem 0x40200000-0x402fffff]
[ 2.504815] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt
[ 2.504821] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[ 2.504830] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded
[ 2.505000] aer 0000:00:01.0:pcie002: service driver aer loaded
[ 2.505347] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt
[ 2.505351] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt
[ 2.505359] pcie_pme 0000:00:03.0:pcie001: service driver pcie_pme loaded
[ 2.505515] aer 0000:00:03.0:pcie002: service driver aer loaded
[ 2.505613] pci 0000:01:00.0: enabling device (0000 ā†’ 0002)
[ 3.731958] ehci-pci: EHCI PCI platform driver
[ 3.731998] ohci-pci: OHCI PCI platform driver
[ 9.425767] pci 0000:01:00.0: xHCI HW not ready after 5 sec (HC bug?) status = 0x801
[ 9.433571] sysfs: cannot create duplicate filename ā€˜/devices/10003000.pcie-controller/pci0000:00/0000:00:01.0/0000:01:00.0/configā€™
[ 9.450218] Workqueue: events pcie_delayed_detect
[ 9.450281] [] pci_create_sysfs_dev_files+0x4c/0x318
[ 9.450285] [] pci_bus_add_device+0x38/0xb0
[ 9.450287] [] pci_bus_add_devices+0x40/0x90
[ 9.450289] [] pci_bus_add_devices+0x88/0x90
[ 9.450291] [] pcie_delayed_detect+0x96c/0xdf8
[ 49.638454] pci 0000:02:00.0: enabling device (0000 ā†’ 0002)
[ 56.030266] pci 0000:02:00.0: xHCI HW not ready after 5 sec (HC bug?) status = 0x801
[ 56.038044] sysfs: cannot create duplicate filename ā€˜/devices/10003000.pcie-controller/pci0000:00/0000:00:03.0/0000:02:00.0/configā€™
[ 56.066440] Workqueue: events pcie_delayed_detect
[ 56.071135] [] pci_create_sysfs_dev_files+0x4c/0x318
[ 56.071139] [] pci_bus_add_device+0x38/0xb0
[ 56.071142] [] pci_bus_add_devices+0x40/0x90
[ 56.071145] [] pci_bus_add_devices+0x88/0x90
[ 56.071147] [] pcie_delayed_detect+0x96c/0xdf8

By the way, can this prove that there is no problem with my hardware circuit design?

@GingKo,

There was an issue with file uploads in the forums that was resolved this morning. Please try uploading now.

Thanks,
Tom

1 Like

dts&dmesg.zip (142.1 KB)

Hi,
Is it still a dts issue this time?

[   56.038044] sysfs: cannot create duplicate filename '/devices/10003000.pcie-controller/pci0000:00/0000:00:03.0/0000:02:00.0/config'
...
[   56.071124] [<ffffff8008473440>] sysfs_warn_dup+0x68/0x88

I couldnā€™t tell you what the duplicate is, but it thinks you have two devices in the same PCIe slot. In theory part of the problem could be something passed by device tree. Your ā€œmissingā€ device probably has some duplication with another device.

Thanks for your answer, I will continue to modify the dts file next.

I modified the dts file, but now the first error appears again: pcie0 and pcie2 are not recognized. This time I saw that dmesg shows that there is no duplicate device. The strangest thing is that I canā€™t reproduce the previous error-even if I use the previous dtb backup, it cannot appear that only pcie1 is available, the other two are unavailable, and the error ā€œsysfs: cannot create duplicate filenameā€ is reported.

dts_new.zip (122.2 KB)

The attached dmesg part of the zip file was empty, so there is no way to see any message about PCIe during boot (youā€™d really want a serial console boot log so it shows content prior to reaching the Linux kernelā€¦this is superior to a dmesg when you need to know about early setup of a device).

Hint on lspci: It needs sudo to display some of the details of increased verbosity levels.

I donā€™t know enough about PCIe lane setup to help test further, but if you do have two PCIe devices with one not showing up, then there is a failure prior to the USB part of the device tree. If you made any kind of edit specific to PCIe lane mapping, then you might want to submit just a clip of the before and after for that section of the dts.

Hiļ¼Œ
This is the complete dmesg output document and sudo lspci -vvv document.

You are right, but unfortunately I did not design a serial port that uses UART0 for debugging. I will pay attention to it in future designs.

By the way, I am a little skeptical that this is a signal quality problemļ¼Œif there is no problem with my schematic design.
dts_new_2.zip (141.8 KB)

Just to emphasize, I donā€™t think I can be much help at this point, but I do see this which sticks out (several times):

[    3.624112] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[    3.626162] tegra-pcie 10003000.pcie-controller: link 2 down, ignoring

Followed later by:

[   10.555032] pci 0000:01:00.0: xHCI HW not ready after 5 sec (HC bug?) status = 0x801

I donā€™t know what device tree entry might be related to that, but it seems like it was trying to bring the link up. Most of the custom carrier board cases where something just does not come up, but should otherwise be ok, fall into the device category of failure.

The lspci wonā€™t show something if the link is down. The bridge and the USB controller which shows up have no errors, so I would expect that if the device tree were correct, then your other device would also show up. Someone else will need to look at the device tree since I have not worked on custom carrier boards.