PCIe, a riser and a Nano

[serg_soft]

Just tried to connect a Nano board to some external PCIe devices using a couple of adapters for mini PCIe slots finally came to a riser.

This configuration with an Nvideo-based video card did not work to me:

dmesg | grep pci

[ 0.956082] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.3, lane = pcie-0, function = pcie-x1

[ 0.956196] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.4, lane = pcie-1, function = pcie-x4

[ 0.956309] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.5, lane = pcie-2, function = pcie-x4

[ 0.956425] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.6, lane = pcie-3, function = pcie-x4

[ 0.956532] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.7, lane = pcie-4, function = pcie-x4

[ 0.956640] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.8, lane = pcie-5, function = xusb

[ 0.956748] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.9, lane = pcie-6, function = xusb

[ 0.966351] tegra-pcie 1003000.pcie: 4x1, 1x1 configuration

[ 0.967644] tegra-pcie 1003000.pcie: PCIE: Enable power rails

[ 0.968009] tegra-pcie 1003000.pcie: probing port 0, using 4 lanes

[ 0.972071] tegra-pcie 1003000.pcie: probing port 1, using 1 lanes

[ 1.898001] tegra-pcie 1003000.pcie: link 0 down, retrying

[ 2.301868] tegra-pcie 1003000.pcie: link 0 down, retrying

[ 2.706181] tegra-pcie 1003000.pcie: link 0 down, retrying

[ 2.708197] tegra-pcie 1003000.pcie: link 0 down, ignoring

[ 2.814410] tegra-pcie 1003000.pcie: PCI host bridge to bus 0000:00

Actually, I tried two cards with the same output.

On the other hand, the TP-link adapter (which is also on the photo) was detected and available via lspci.

Just wanted to know the reason why.

[linuxdev]

This is not a full answer, but I will point out that the GPU built in to the Nano is an integrated GPU (iGPU) directly wired to the memory controller. External discrete GPUs (dGPU) use software which goes through the PCI bus to talk to the GPU. The drivers for the two are distinct and different. The driver which the Nano comes with will not work for a PCIe GPU, and possibly dual install of that driver plus an arm64/aarch64 PCIe driver might conflict. Regardless, it probably wouldn’t work unless you have both drivers. I don’t know of a way to install or work with the dGPU driver.

[serg_soft]

Yes, the drivers will be another issue.
I also have another embedded ARM board with PCIe, it connects but fails to memory map, probably it is impossible due to the memory model, do you know?

In the case of Nano, it fails at the first stage (physical link), an x1 card was detected, two x16 wasn’t, any ideas why and how to obtain more diagnostic info ?

[linuxdev]

I’m thinking someone else will have to answer with knowledge of the driver, but can you post the exact driver you are using, e.g., the exact package name which provides this? And also, can we verify the log snippet you just posted is for the dGPU and not the Jetson’s iGPU?

Generally speaking, for PCI devices (and the iGPU is not PCI), one can see devices with “lspci”. The result starts with a slot ID, e.g., something like “00:00.0” in format. You can limit a query to a specific device with that. For example, this would query just the one device in slot “00:00.0”:
lspci -s 00:00.0

To get a fully verbose listing, and to log it, I’ll suggest you find the log for that slot like this (adjust the actual slot ID which might differ, and use sudo since it is mandatory for more detailed information):
sudo lspci -s 00:00.0 -vvv 2>&1 | tee log_pci.txt

The above would provide the most verbose PCI information possible for just that slot, and create a log you can attach in the forum.

Also, you might mention if the iGPU is functioning correctly when the dGPU is present.

[fchkjwlsq]

@serg_soft

I also have another embedded ARM board with PCIe, it connects but fails to memory map, probably it is impossible due to the memory model, do you know?

Look for Jeff Geerling on Youtube. He messes with Raspberry Pi’s in his videos on a regular basis and has tried to get standard PCIe graphic cards from AMD and NVidia working on the PCIe x1 of a CM4. He failed.
Reasons:

• Kernel arch-dependend (amd64) code.
• BAR issues (memory windows not big enough, total mappable addres space not big enough)
• interrupt issues
• closed source drivers

If you need an external GPU on an ARM system, then look for an LX2160 based board from www.solid-run.com They claim to have AMD GPUs working in theit ARM64 based boards. NVidia-based GPUs are difficult because of the closed source drivers and lack of microcode for these cards.

Most probably your idea simply won’t work.

[serg_soft]

Yes, I understand about lspci. I connect a dGPU, the iGPU is not visible since it is not PCIe-related.

The problem is that the external devices are not well connected using such risers for Nano (they work in other cases, probably it is the issue of susceptibility to such levels of electrical signals). It tries 4 times and skips the device (the log in the first message).

I tried several and the only card I can connect to Nano is an old 128 (or 256) mb ATI x600:)

But seems the memory BAR requested by the card is too huge.

[ 1.520578] pci 0000:01:00.1: [1002:3e70] type 00 class 0x038000

[ 1.520604] pci 0000:01:00.1: reg 0x10: [mem 0x00000000-0x0000ffff]

[ 1.520746] pci 0000:01:00.1: supports D1 D2

[ 1.520950] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with ‘pcie_aspm=force’

[ 1.520991] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01

[ 1.521160] pci 0000:02:00.0: [10ec:8168] type 00 class 0x020000

[ 1.521188] pci 0000:02:00.0: reg 0x10: [io 0x0000-0x00ff]

[ 1.521222] pci 0000:02:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]

[ 1.521245] pci 0000:02:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]

[ 1.521403] pci 0000:02:00.0: supports D1 D2

[ 1.521409] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold

[ 1.530350] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02

[ 1.530426] pci 0000:00:01.0: BAR 14: no space for [mem size 0x18000000]

[ 1.530433] pci 0000:00:01.0: BAR 14: failed to assign [mem size 0x18000000]

[ 1.530439] pci 0000:00:02.0: BAR 14: assigned [mem 0x13000000-0x130fffff]

[ 1.530443] pci 0000:00:01.0: BAR 13: assigned [io 0x1000-0x1fff]

[ 1.530448] pci 0000:00:02.0: BAR 13: assigned [io 0x2000-0x2fff]

[ 1.530457] pci 0000:01:00.0: BAR 0: no space for [mem size 0x10000000 pref]

[ 1.530462] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x10000000 pref]

[ 1.530467] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00020000 pref]

[ 1.530471] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00020000 pref]

[ 1.530475] pci 0000:01:00.0: BAR 2: no space for [mem size 0x00010000]

[ 1.530478] pci 0000:01:00.0: BAR 2: failed to assign [mem size 0x00010000]

[ 1.530483] pci 0000:01:00.1: BAR 0: no space for [mem size 0x00010000]

[ 1.530486] pci 0000:01:00.1: BAR 0: failed to assign [mem size 0x00010000]

[ 1.530490] pci 0000:01:00.0: BAR 1: assigned [io 0x1000-0x10ff]

[ 1.530501] pci 0000:00:01.0: PCI bridge to [bus 01]

[ 1.530508] pci 0000:00:01.0: bridge window [io 0x1000-0x1fff]

[ 1.530522] pci 0000:02:00.0: BAR 4: assigned [mem 0x13000000-0x13003fff 64bit]

[ 1.530542] pci 0000:02:00.0: BAR 2: assigned [mem 0x13004000-0x13004fff 64bit]

[ 1.530560] pci 0000:02:00.0: BAR 0: assigned [io 0x2000-0x20ff]

[ 1.530568] pci 0000:00:02.0: PCI bridge to [bus 02]

[ 1.530572] pci 0000:00:02.0: bridge window [io 0x2000-0x2fff]

[ 1.530578] pci 0000:00:02.0: bridge window [mem 0x13000000-0x130fffff]

[ 1.531032] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt

[ 1.531036] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt

[ 1.531039] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt

[ 1.531046] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded

[ 1.531163] aer 0000:00:01.0:pcie002: service driver aer loaded

[ 1.531379] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt

[ 1.531383] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt

[ 1.531389] pcie_pme 0000:00:02.0:pcie001: service driver pcie_pme loaded

[ 1.531486] aer 0000:00:02.0:pcie002: service driver aer loaded

[ 4.569972] ehci-pci: EHCI PCI platform driver

[ 4.580674] ohci-pci: OHCI PCI platform driver

$ lspci

00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1)

00:02.0 PCI bridge: NVIDIA Corporation Device 0faf (rev a1)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV380 [Radeon X600]

01:00.1 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] RV380 [Radeon X600] (Secondary)

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

[linuxdev]

It would be difficult to say for certain, but riser cards do often make signal requirements more difficult. It could be an actual driver issue, e.g., it doesn’t understand something about the Jetson’s design, but signal quality issues are quite common. It would be quite difficult to get a better answer without someone having the same model of Jetson, the same video card and riser, and perhaps even rather expensive PCIe signal testing equipment (and perhaps the source code of the driver itself). I wouldn’t even have a chance at that.

The best chance is if you can state the exact driver you’ve installed, and someone knowing that driver knows how to get verbose logging (or alternatively, know which “/sys” files to consult).

[fchkjwlsq]

Home

[linuxdev]

That URL does not say which specific GPU and driver you are using. FYI, it is possible that a discrete GPU driver for a PCIe RPi could be compatible with a Jetson, but you’d need more details. Also mention if the GPU is externally powered (I have doubts about the PCIe bus being able to power a discrete GPU).

[serg_soft]

Very thanks for the link, for each card there is a description of the sequence of actions, I will try some.

[serg_soft]

The riser provides power for all PCIe power lines, only data lines from Nano are used.

[linuxdev]

I wouldn’t know what else to check in that case. It is possible that a driver for the dGPU is altering the iGPU driver or configuration since it does not understand how it works (the dGPU driver would have no PCIe query method to know what else is there).