PCIe C5 as RP and x1 NIC card as EP on AGX Orin Dev kit

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi Nvidia support team,

I am having a AGX orin EV kit and trying to enable x1 lane NIC card by connecting it to x16 standard PCIe slot. I verified the NIC card with another linux PC, and its detected with lspci, but failing with Orin board. Any hints on this would be helpful.

HW used: Nvidia AGX orin 32GB dev kit
SW used: Jetpack used is 6.2.1 where the kernel release is 36.4.4.

I have measured the following lines on the x16 standard PCIe connector at Orin side.
Side B:
Pin 1, 2, 3 - 12V during link up process or when bind command is given
Pin 10 - 3.3V aux - always on
Pin 17 - PRSNT2# - 3.3V when NIC is not connected, Gnd when NIC is connected.

Side A:
Pin 1: PRSNT1# - always low
Pin 2 & 3 - 12V during link up process or when bind command is given
Pin 11 - PERST# - 3.3V during link up process or when bind command is given
Pin 13 & 14 - REFCLK+ and REFCLK- meaasured as 100MHz during link up process or when bind command is given

The voltage and clock lines seem to be okay. Kindly suggest if any further lines can be measured to rule out HW issue.

Attached the dmesg logs for the pcie C5 port(0x141a0000):

11.347022] tegra194-pcie 141a0000.pcie: host bridge /bus@0/pcie@141a0000 ranges:
[ 11.347048] tegra194-pcie 141a0000.pcie: MEM 0x2800000000..0x2b27ffffff → 0x2800000000
[ 11.347054] tegra194-pcie 141a0000.pcie: MEM 0x2b28000000..0x2b2fffffff → 0x0040000000
[ 11.347057] tegra194-pcie 141a0000.pcie: IO 0x003a100000..0x003a1fffff → 0x003a100000
[ 11.347402] tegra194-pcie 141a0000.pcie: iATU unroll: enabled
[ 11.347406] tegra194-pcie 141a0000.pcie: Detected iATU regions: 8 outbound, 2 inbound
[ 12.446510] tegra194-pcie 141a0000.pcie: Phy link never came up
[ 13.426366] tegra194-pcie 141a0000.pcie: Phy link never came up
[ 13.426434] tegra194-pcie 141a0000.pcie: PCI host bridge to bus 0005:00
[ 13.426437] pci_bus 0005:00: root bus resource [io 0x200000-0x2fffff] (bus address [0x3a100000-0x3a1fffff])
[ 13.426441] pci_bus 0005:00: root bus resource [mem 0x2b28000000-0x2b2fffffff] (bus address [0x40000000-0x47ffffff])
[ 13.426444] pci_bus 0005:00: root bus resource [bus 00-ff]
[ 13.426446] pci_bus 0005:00: root bus resource [mem 0x2800000000-0x2b27ffffff pref]
[ 13.426495] pci 0005:00:00.0: [10de:229a] type 01 class 0x060400
[ 13.426655] pci 0005:00:00.0: PME# supported from D0 D3hot
[ 13.432667] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[ 13.432821] pcieport 0005:00:00.0: Adding to iommu group 54
[ 13.432943] pcieport 0005:00:00.0: PME: Signaling with IRQ 204
[ 13.433379] pcieport 0005:00:00.0: AER: enabled with IRQ 204

4

Is this a true NV devkit or a custom board?

5

Hi @WayneWWW , yes, it’s a NV devkit, not a custom board.

6

I don’t think it is related to basic pinmux or voltage configuration on this as C5 has everything enabled on devkit already.

Please check if limiting the card speed to gen 1 in device tree would help or not.

adding “max-link-speed = <1>;” property to your pcie controller

7

Hi @WayneWWW ,

I have updated the max link speed as suggested. The behavior is still same.

orin-nvidia@orin-nvidia:/proc/device-tree/bus@0/pcie@141a0000$ xxd max-link-speed
00000000: 0000 0001 …

Also, could you confirm if HPD is supported in C5?

8

Hi @WayneWWW ,

I have some update. I connected Intel 10G Ethernet Card X550-T2V2 and it got detected in Orin devkit. Following are the list of cards I have tried out, and the status. All these cards are detected in a Linux PC without any issues. Could you please post if you have some hints why first two cards are not getting detected?

PCIe card Width Gen & Speed Detection in Orin dev kit
Realtek NIC card x1 Gen1 (2.5GT/s) Not detected
SSD card adapter x4 Gen4 (16GT/s) Not detected
Intel 10G Ethernet Card X550-T2V2 x4 Gen3 (8GT/s) Detected
9

I have updated the max link speed as suggested. The behavior is still same.

Check if those devices which are still working are all working under gen1. This is to confirm the change is taking effect.

Also, could you confirm if HPD is supported in C5?

No, it is not supported.

10

Hi @WayneWWW ,

Please find the lspci verbose logs with the working device connected to C5.
Link capability of the bridge has updated with speed of 2.5GT/s(Gen1). The end device downgraded its speed accordingly and the link got established.
So the device tree update seems to be reflecting rightly.

0005:00:00.0 PCI bridge: NVIDIA Corporation Device 229a (rev a1) (prog-if 00 [Normal decode])

                LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <64us
                        ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+

                LnkSta: Speed 2.5GT/s (ok), Width x4 (downgraded)
                        TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-


0005:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkSta: Speed 2.5GT/s (downgraded), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
11

Then to check why some devices are not able to detect, need to use PCIe analyzer to get a LA trace.

No other method that might precisely tell what goes wrong.