The ethernet interface over PCIe(endpoint mode) not work

[hermes_wu]

Hello,

I used two Jetson AGX Xavier DevKit and connected by PCIe, and follow the guide below:
https://docs.nvidia.com/jetson/archives/r35.2.1/DeveloperGuide/text/SD/Communications/PcieEndpointMode.html

I tried to bring up the ethernet interface over PCIe, and the eth interface has appeared both on the ep and rc sides. But they can’t communicate by the eth interface, even the static IP address assigned.

Jetson Linux : 35.2.1

---- EP log ----

root@tegra-ubuntu:/sys/kernel/config/pci_ep# dmesg | grep -i pci
[    0.739560] PCI: CLS 0 bytes, default 64
[    2.772446] ohci-pci: OHCI PCI platform driver
[    4.425390] tegra194-pcie 141a0000.pcie_ep: Adding to iommu group 7
[    4.432117] tegra194-pcie 141a0000.pcie_ep: Failed to get PERST GPIO: -517
[    4.432131] tegra194-pcie 141a0000.pcie_ep: Failed to parse device tree: -517
[    4.432679] tegra194-pcie 14180000.pcie: Adding to iommu group 8
[    4.441612] tegra194-pcie 14180000.pcie: host bridge /pcie@14180000 ranges:
[    4.443151] tegra194-pcie 14180000.pcie:       IO 0x0038100000..0x00381fffff -> 0x0038100000
[    4.451865] tegra194-pcie 14180000.pcie:      MEM 0x1800000000..0x1b3fffffff -> 0x1800000000
[    4.460116] tegra194-pcie 14180000.pcie:      MEM 0x1b40000000..0x1bffffffff -> 0x0040000000
[    5.567821] tegra194-pcie 14180000.pcie: Phy link never came up
[    5.568360] tegra194-pcie 14180000.pcie: PCI host bridge to bus 0000:00
[    5.568574] pci_bus 0000:00: root bus resource [bus 00-ff]
[    5.568787] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[    5.569149] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1b3fffffff pref]
[    5.569388] pci_bus 0000:00: root bus resource [mem 0x1b40000000-0x1bffffffff] (bus address [0x40000000-0xffffffff])
[    5.569914] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[    5.570386] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    5.597770] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    5.598121] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    5.600490] pcieport 0000:00:00.0: Adding to iommu group 8
[    5.602276] pcieport 0000:00:00.0: PME: Signaling with IRQ 25
[    5.603628] pcieport 0000:00:00.0: AER: enabled with IRQ 25
[    5.605418] pci_bus 0000:01: busn_res: [bus 01-ff] is released
[    5.606399] pci 0000:00:00.0: Removing from iommu group 8
[    5.606717] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[    5.611223] tegra194-pcie 14100000.pcie: Adding to iommu group 9
[    5.616312] tegra194-pcie 14100000.pcie: host bridge /pcie@14100000 ranges:
[    5.616686] tegra194-pcie 14100000.pcie:       IO 0x0030100000..0x00301fffff -> 0x0030100000
[    5.617080] tegra194-pcie 14100000.pcie:      MEM 0x1200000000..0x122fffffff -> 0x1200000000
[    5.617459] tegra194-pcie 14100000.pcie:      MEM 0x1230000000..0x123fffffff -> 0x0040000000
[    5.731821] tegra194-pcie 14100000.pcie: Link up
[    5.743472] tegra194-pcie 14100000.pcie: PCI host bridge to bus 0001:00
[    5.743693] pci_bus 0001:00: root bus resource [bus 00-ff]
[    5.743921] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x30100000-0x301fffff])
[    5.744229] pci_bus 0001:00: root bus resource [mem 0x1200000000-0x122fffffff pref]
[    5.744485] pci_bus 0001:00: root bus resource [mem 0x1230000000-0x123fffffff] (bus address [0x40000000-0x4fffffff])
[    5.744901] pci 0001:00:00.0: [10de:1ad2] type 01 class 0x060400
[    5.745361] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[    5.758074] pci 0001:01:00.0: [1b4b:9171] type 00 class 0x010601
[    5.758490] pci 0001:01:00.0: reg 0x10: [io  0x8000-0x8007]
[    5.758795] pci 0001:01:00.0: reg 0x14: [io  0x8040-0x8043]
[    5.759064] pci 0001:01:00.0: reg 0x18: [io  0x8100-0x8107]
[    5.759341] pci 0001:01:00.0: reg 0x1c: [io  0x8140-0x8143]
[    5.759610] pci 0001:01:00.0: reg 0x20: [io  0x800000-0x80000f]
[    5.759983] pci 0001:01:00.0: reg 0x24: [mem 0x00900000-0x009001ff]
[    5.760280] pci 0001:01:00.0: reg 0x30: [mem 0xd0000000-0xd000ffff pref]
[    5.761093] pci 0001:01:00.0: PME# supported from D3hot
[    5.783190] pci 0001:00:00.0: BAR 14: assigned [mem 0x1230000000-0x12300fffff]
[    5.783567] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    5.783863] pci 0001:01:00.0: BAR 6: assigned [mem 0x1230000000-0x123000ffff pref]
[    5.784115] pci 0001:01:00.0: BAR 5: assigned [mem 0x1230010000-0x12300101ff]
[    5.786058] pci 0001:01:00.0: BAR 4: assigned [io  0x100000-0x10000f]
[    5.792583] pci 0001:01:00.0: BAR 0: assigned [io  0x100010-0x100017]
[    5.799154] pci 0001:01:00.0: BAR 2: assigned [io  0x100018-0x10001f]
[    5.805403] pci 0001:01:00.0: BAR 1: assigned [io  0x100020-0x100023]
[    5.811787] pci 0001:01:00.0: BAR 3: assigned [io  0x100024-0x100027]
[    5.818311] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    5.823418] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    5.830233] pci 0001:00:00.0:   bridge window [mem 0x1230000000-0x12300fffff]
[    5.837565] pci 0001:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    5.845997] pci 0001:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  512
[    5.855667] pcieport 0001:00:00.0: Adding to iommu group 9
[    5.860656] pcieport 0001:00:00.0: PME: Signaling with IRQ 27
[    5.866407] pcieport 0001:00:00.0: AER: enabled with IRQ 27
[    5.873162] tegra194-pcie 14140000.pcie: Adding to iommu group 10
[    5.880330] tegra194-pcie 14140000.pcie: host bridge /pcie@14140000 ranges:
[    5.884542] tegra194-pcie 14140000.pcie:       IO 0x0034100000..0x00341fffff -> 0x0034100000
[    5.893093] tegra194-pcie 14140000.pcie:      MEM 0x1280000000..0x12afffffff -> 0x1280000000
[    5.901650] tegra194-pcie 14140000.pcie:      MEM 0x12b0000000..0x12bfffffff -> 0x0040000000
[    7.003886] tegra194-pcie 14140000.pcie: Phy link never came up
[    7.004345] tegra194-pcie 14140000.pcie: PCI host bridge to bus 0003:00
[    7.004577] pci_bus 0003:00: root bus resource [bus 00-ff]
[    7.004740] pci_bus 0003:00: root bus resource [io  0x200000-0x2fffff] (bus address [0x34100000-0x341fffff])
[    7.005088] pci_bus 0003:00: root bus resource [mem 0x1280000000-0x12afffffff pref]
[    7.005315] pci_bus 0003:00: root bus resource [mem 0x12b0000000-0x12bfffffff] (bus address [0x40000000-0x4fffffff])
[    7.005790] pci 0003:00:00.0: [10de:1ad2] type 01 class 0x060400
[    7.006290] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[    7.028735] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[    7.028984] pci 0003:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    7.030214] pcieport 0003:00:00.0: Adding to iommu group 10
[    7.031177] pcieport 0003:00:00.0: PME: Signaling with IRQ 29
[    7.032201] pcieport 0003:00:00.0: AER: enabled with IRQ 29
[    7.033484] pci_bus 0003:01: busn_res: [bus 01-ff] is released
[    7.034248] pci 0003:00:00.0: Removing from iommu group 10
[    7.034514] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[   32.037061] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[   32.165174] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[   32.288201] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[   32.396980] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[   32.413002] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up
[   39.904020] vdd-3v3-pcie: disabling
[   39.904049] vdd-12v-pcie: disabling
[   41.508833] tegra194-pcie 141a0000.pcie_ep: LTSSM state: 0x2018 timeout: -110
root@tegra-ubuntu:/sys/kernel/config/pci_ep#

---- RC log ----

root@tegra-ubuntu:/home/denso# dmesg | grep -i pci
[    0.730468] PCI: CLS 0 bytes, default 64
[    2.885237] ohci-pci: OHCI PCI platform driver
[    4.401785] tegra194-pcie 14180000.pcie: Adding to iommu group 7
[    4.413108] tegra194-pcie 14180000.pcie: host bridge /pcie@14180000 ranges:
[    4.414029] tegra194-pcie 14180000.pcie:       IO 0x0038100000..0x00381fffff -> 0x0038100000
[    4.422787] tegra194-pcie 14180000.pcie:      MEM 0x1800000000..0x1b3fffffff -> 0x1800000000
[    4.430912] tegra194-pcie 14180000.pcie:      MEM 0x1b40000000..0x1bffffffff -> 0x0040000000
[    5.546824] tegra194-pcie 14180000.pcie: Phy link never came up
[    5.547340] tegra194-pcie 14180000.pcie: PCI host bridge to bus 0000:00
[    5.547618] pci_bus 0000:00: root bus resource [bus 00-ff]
[    5.547787] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0x38100000-0x381fffff])
[    5.548084] pci_bus 0000:00: root bus resource [mem 0x1800000000-0x1b3fffffff pref]
[    5.548380] pci_bus 0000:00: root bus resource [mem 0x1b40000000-0x1bffffffff] (bus address [0x40000000-0xffffffff])
[    5.548856] pci 0000:00:00.0: [10de:1ad0] type 01 class 0x060400
[    5.549352] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
[    5.583231] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    5.583497] pci 0000:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    5.585666] pcieport 0000:00:00.0: Adding to iommu group 7
[    5.587457] pcieport 0000:00:00.0: PME: Signaling with IRQ 24
[    5.589531] pcieport 0000:00:00.0: AER: enabled with IRQ 24
[    5.591591] pci_bus 0000:01: busn_res: [bus 01-ff] is released
[    5.592575] pci 0000:00:00.0: Removing from iommu group 7
[    5.592894] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[    5.596984] tegra194-pcie 14100000.pcie: Adding to iommu group 8
[    5.602824] tegra194-pcie 14100000.pcie: host bridge /pcie@14100000 ranges:
[    5.603193] tegra194-pcie 14100000.pcie:       IO 0x0030100000..0x00301fffff -> 0x0030100000
[    5.603547] tegra194-pcie 14100000.pcie:      MEM 0x1200000000..0x122fffffff -> 0x1200000000
[    5.603895] tegra194-pcie 14100000.pcie:      MEM 0x1230000000..0x123fffffff -> 0x0040000000
[    5.709981] tegra194-pcie 14100000.pcie: Link up
[    5.721950] tegra194-pcie 14100000.pcie: PCI host bridge to bus 0001:00
[    5.722154] pci_bus 0001:00: root bus resource [bus 00-ff]
[    5.722323] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x30100000-0x301fffff])
[    5.722603] pci_bus 0001:00: root bus resource [mem 0x1200000000-0x122fffffff pref]
[    5.722827] pci_bus 0001:00: root bus resource [mem 0x1230000000-0x123fffffff] (bus address [0x40000000-0x4fffffff])
[    5.723214] pci 0001:00:00.0: [10de:1ad2] type 01 class 0x060400
[    5.723613] pci 0001:00:00.0: PME# supported from D0 D3hot D3cold
[    5.737284] pci 0001:01:00.0: [1b4b:9171] type 00 class 0x010601
[    5.737704] pci 0001:01:00.0: reg 0x10: [io  0x8000-0x8007]
[    5.738187] pci 0001:01:00.0: reg 0x14: [io  0x8040-0x8043]
[    5.738451] pci 0001:01:00.0: reg 0x18: [io  0x8100-0x8107]
[    5.738721] pci 0001:01:00.0: reg 0x1c: [io  0x8140-0x8143]
[    5.739028] pci 0001:01:00.0: reg 0x20: [io  0x800000-0x80000f]
[    5.739341] pci 0001:01:00.0: reg 0x24: [mem 0x00900000-0x009001ff]
[    5.739621] pci 0001:01:00.0: reg 0x30: [mem 0xd0000000-0xd000ffff pref]
[    5.740348] pci 0001:01:00.0: PME# supported from D3hot
[    5.763463] pci 0001:00:00.0: BAR 14: assigned [mem 0x1230000000-0x12300fffff]
[    5.763802] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    5.764067] pci 0001:01:00.0: BAR 6: assigned [mem 0x1230000000-0x123000ffff pref]
[    5.764365] pci 0001:01:00.0: BAR 5: assigned [mem 0x1230010000-0x12300101ff]
[    5.764626] pci 0001:01:00.0: BAR 4: assigned [io  0x100000-0x10000f]
[    5.771054] pci 0001:01:00.0: BAR 0: assigned [io  0x100010-0x100017]
[    5.777347] pci 0001:01:00.0: BAR 2: assigned [io  0x100018-0x10001f]
[    5.783884] pci 0001:01:00.0: BAR 1: assigned [io  0x100020-0x100023]
[    5.790447] pci 0001:01:00.0: BAR 3: assigned [io  0x100024-0x100027]
[    5.796923] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    5.802185] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    5.808463] pci 0001:00:00.0:   bridge window [mem 0x1230000000-0x12300fffff]
[    5.815778] pci 0001:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    5.824698] pci 0001:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  512
[    5.834086] pcieport 0001:00:00.0: Adding to iommu group 8
[    5.839130] pcieport 0001:00:00.0: PME: Signaling with IRQ 26
[    5.845332] pcieport 0001:00:00.0: AER: enabled with IRQ 26
[    5.851682] tegra194-pcie 14140000.pcie: Adding to iommu group 9
[    5.859402] tegra194-pcie 14140000.pcie: host bridge /pcie@14140000 ranges:
[    5.863035] tegra194-pcie 14140000.pcie:       IO 0x0034100000..0x00341fffff -> 0x0034100000
[    5.871676] tegra194-pcie 14140000.pcie:      MEM 0x1280000000..0x12afffffff -> 0x1280000000
[    5.880175] tegra194-pcie 14140000.pcie:      MEM 0x12b0000000..0x12bfffffff -> 0x0040000000
[    6.994400] tegra194-pcie 14140000.pcie: Phy link never came up
[    6.994890] tegra194-pcie 14140000.pcie: PCI host bridge to bus 0003:00
[    6.995065] pci_bus 0003:00: root bus resource [bus 00-ff]
[    6.995186] pci_bus 0003:00: root bus resource [io  0x200000-0x2fffff] (bus address [0x34100000-0x341fffff])
[    6.995434] pci_bus 0003:00: root bus resource [mem 0x1280000000-0x12afffffff pref]
[    6.995601] pci_bus 0003:00: root bus resource [mem 0x12b0000000-0x12bfffffff] (bus address [0x40000000-0x4fffffff])
[    6.995920] pci 0003:00:00.0: [10de:1ad2] type 01 class 0x060400
[    6.996249] pci 0003:00:00.0: PME# supported from D0 D3hot D3cold
[    7.012908] pci 0003:00:00.0: PCI bridge to [bus 01-ff]
[    7.013119] pci 0003:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    7.014463] pcieport 0003:00:00.0: Adding to iommu group 9
[    7.015403] pcieport 0003:00:00.0: PME: Signaling with IRQ 28
[    7.016442] pcieport 0003:00:00.0: AER: enabled with IRQ 28
[    7.017385] pci_bus 0003:01: busn_res: [bus 01-ff] is released
[    7.018096] pci 0003:00:00.0: Removing from iommu group 9
[    7.018301] pci_bus 0003:00: busn_res: [bus 00-ff] is released
[    7.020284] tegra194-pcie 141a0000.pcie: Adding to iommu group 10
[    7.021702] tegra194-pcie 141a0000.pcie: Failed to get slot regulators: -517
[    7.667526] tegra194-pcie 141a0000.pcie: Failed to get slot regulators: -517
[    8.102274] tegra194-pcie 141a0000.pcie: host bridge /pcie@141a0000 ranges:
[    8.102304] tegra194-pcie 141a0000.pcie:       IO 0x003a100000..0x003a1fffff -> 0x003a100000
[    8.102340] tegra194-pcie 141a0000.pcie:      MEM 0x1c00000000..0x1f3fffffff -> 0x1c00000000
[    8.102349] tegra194-pcie 141a0000.pcie:      MEM 0x1f40000000..0x1fffffffff -> 0x0040000000
[    8.209931] tegra194-pcie 141a0000.pcie: Link up
[    8.216340] tegra194-pcie 141a0000.pcie: PCI host bridge to bus 0005:00
[    8.227179] pci_bus 0005:00: root bus resource [bus 00-ff]
[    8.227187] pci_bus 0005:00: root bus resource [io  0x300000-0x3fffff] (bus address [0x3a100000-0x3a1fffff])
[    8.227191] pci_bus 0005:00: root bus resource [mem 0x1c00000000-0x1f3fffffff pref]
[    8.227213] pci_bus 0005:00: root bus resource [mem 0x1f40000000-0x1fffffffff] (bus address [0x40000000-0xffffffff])
[    8.242452] pci 0005:00:00.0: [10de:1ad0] type 01 class 0x060400
[    8.255170] pci 0005:00:00.0: PME# supported from D0 D3hot D3cold
[    8.281071] pci 0005:01:00.0: [10de:2296] type 00 class 0x028000
[    8.315722] pci 0005:01:00.0: reg 0x10: [mem 0x00000000-0x003fffff]
[    8.321620] pci 0005:01:00.0: reg 0x18: [mem 0x00000000-0x0001ffff 64bit pref]
[    8.329184] pci 0005:01:00.0: reg 0x20: [mem 0x00000000-0x000fffff 64bit]
[    8.336314] pci 0005:01:00.0: PME# supported from D0 D3hot D3cold
[    8.342482] pci 0005:01:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0005:00:00.0 (capable of 126.024 Gb/s with 16.0 GT/s PCIe x8 link)
[    8.363680] pci 0005:00:00.0: BAR 14: assigned [mem 0x1f40000000-0x1f405fffff]
[    8.364379] pci 0005:00:00.0: BAR 15: assigned [mem 0x1c00000000-0x1c000fffff 64bit pref]
[    8.372703] pci 0005:01:00.0: BAR 0: assigned [mem 0x1f40000000-0x1f403fffff]
[    8.380063] pci 0005:01:00.0: BAR 4: assigned [mem 0x1f40400000-0x1f404fffff 64bit]
[    8.387784] pci 0005:01:00.0: BAR 2: assigned [mem 0x1c00000000-0x1c0001ffff 64bit pref]
[    8.395595] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[    8.401047] pci 0005:00:00.0:   bridge window [mem 0x1f40000000-0x1f405fffff]
[    8.408204] pci 0005:00:00.0:   bridge window [mem 0x1c00000000-0x1c000fffff 64bit pref]
[    8.416350] pci 0005:00:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.424612] pci 0005:01:00.0: Max Payload Size set to  256/ 256 (was  256), Max Read Rq  512
[    8.433981] pcieport 0005:00:00.0: Adding to iommu group 10
[    8.439163] pcieport 0005:00:00.0: PME: Signaling with IRQ 30
[    8.444777] pcieport 0005:00:00.0: AER: enabled with IRQ 30
root@tegra-ubuntu:/home/denso#

thanks,
Hermes Wu

[WayneWWW]

May I review the whole steps you’ve done here to enable EP mode?

[hermes_wu]

Hi WayneWWW,

I tried to enable the EP mode following the steps below:

1. boot the EP device and enter commands to configure/enable EP mode

2. boot the RC device and enter lspci command to verify that PCIe link is up

3. Bring up both interfaces to establish an Ethernet link between the systems and assign the interfaces static IP addresses
EP side:

RC side:

4. Try to ping each other
   EP side
   
   
   
   RC side
   
   

[WayneWWW]

No, I mean before you flash the board.

[hermes_wu]

sorry it’s my misunderstanding,

1. patch the file kernel/kernel-5.10/drivers/pci/controller/dwc/pcie-tegra194.c for PCIe virtual network to EP side
2. re-build kernel
3. copy device tree files to Linux_for_Tegra/kernel/dtb/
4. copy kernel image to Linux_for_Tegra/kernel/Image
5. edit the p2972-0000.conf.common file, change ODMDATA from 0x09190000 to 0x09191000 .
6. run the command to reflash device

sudo ./flash.sh jetson-agx-xavier-devkit mmcblk0p1

[WayneWWW]

Which jetpack version is in use?

[hermes_wu]

Jetpack : 5.1
Jetson Linux : 35.2.1

[WayneWWW]

Will it work if doing memory read/write with mmap? The LTSSM error seems from hardware problem.

[hermes_wu]

Yes, I tried ‘pci_epf_nv_test’ with ‘devmem’, and bi-directional data transfer between EP and RC is successful.

[WayneWWW]

Are you saying that you can use this section to write the shared RAM on the EP?

https://docs.nvidia.com/jetson/archives/r35.2.1/DeveloperGuide/text/SD/Communications/PcieEndpointMode.html#accessing-the-shared-ram-on-the-endpoint-device

btw, could you upgrade to 35.3.1? We don’t directly debug on old release.

[hermes_wu]

Yes, I can write and read the share memory on EP side.

OK, I’ll try to upgrade to 35.3.1, and update the result after tested.

[hermes_wu]

Hi,

I have tried version 35.3.1, the test result is the same as version 35.2.1.

1. Read/Write share memory on EP side is successful
2. Virtual network can be established, but can’t communicate(ping).

[hermes_wu]

Hello,

I tried JetPack 4.6.3 + Jetson Linux 32.7.3 (Kernel 4.9.299), and the virtual network can be established, and communication is successful. If JetPack 5.2.1/5.3.1 is a must, Do you have any idea that can solve this issue?

By the way, the performance seems not good, the average bandwidth is about 2.x Gbits (under PCIe x8).

EP side:

RC side:

thanks,

[WayneWWW]

現在的狀況是我們還要去複製看看能不能在jp5碰到你這個問題. 這整個流程可能會很慢.

如果你很急的話麻煩就先用jp4.6.3.

[carolyuu]

Hi hermes_wu,

Please try below command to disable network manager:

$ sudo systemctl stop NetworkManager.service
$ sudo systemctl disable NetworkManager.service

Add the patch on EP and set static IP from both side.
You can reference detail steps from: Jetpack5.0.2 Xavier pcie endpoint mode [repetition] - #9 by WayneWWW

[hermes_wu]

Hi @carolyuu,

I tried to disable the network manager for both EP and RC, but they still can’t ping each other.
Jetpack : 5.1
Jetson Linux : 35.2.1

EP Log:

root@tegra-ubuntu:/home/denso# systemctl status NetworkManager.service
● NetworkManager.service - Network Manager
     Loaded: loaded (/lib/systemd/system/NetworkManager.service; disabled; vend>
     Active: inactive (dead)
       Docs: man:NetworkManager(8)
root@tegra-ubuntu:/home/denso# cd /sys/kernel/config/pci_ep/                   1
ho 16 > functions/pci_epf_tvnet/func1/msi_interrupts_ep# mkdir functions/pci_epf_tvnet/func1 
ln -s functions/pci_epf_tvnet/func1 controllersroot@tegra-ubuntu:/sys/kernel/config/pci_ep# echo 16 > functions/pci_epf_tvnet/func1/msi_interrupts
p/
echo root@tegra-ubuntu:/sys/kernel/config/pci_ep# ln -s functions/pci_epf_tvnet/func1 controllers/141a0000.pcie_ep/
141a0000.pcie_ep/start
root@tegra-ubuntu:/sys/kernel/config/pci_ep# echo 1 > controllers/141a0000.pcie_ep/start
root@tegra-ubuntu:/sys/kernel/config/pci_ep# [   42.849523] pci_epf_tvnet pci_epf_tvnet.0: tvnet_ep_open: PCIe link is not up

root@tegra-ubuntu:/sys/kernel/config/pci_ep# [   51.278141] tegra194-pcie 141a0000.pcie_ep: LTSSM state: 0x2018 timeout: -110

root@tegra-ubuntu:/sys/kernel/config/pci_ep# ifconfig eth1 up
root@tegra-ubuntu:/sys/kernel/config/pci_ep# [  127.283921] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.284143] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.285247] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.285431] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.289252] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.289525] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.536388] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.536599] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.786675] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!
[  127.786905] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #08!!!

root@tegra-ubuntu:/sys/kernel/config/pci_ep# ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 64512
        inet 10.10.10.2  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::a8d5:caff:feb3:b36d  prefixlen 64  scopeid 0x20<link>
        ether aa:d5:ca:b3:b3:6d  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@tegra-ubuntu:/sys/kernel/config/pci_ep# ping 10.10.10.3
PING 10.10.10.3 (10.10.10.3) 56(84) bytes of data.

^C
--- 10.10.10.3 ping statistics ---
33 packets transmitted, 0 received, 100% packet loss, time 32762ms

root@tegra-ubuntu:/sys/kernel/config/pci_ep# 

RC Log:

root@tegra-ubuntu:/home/denso# systemctl status NetworkManager.service
● NetworkManager.service - Network Manager
     Loaded: loaded (/lib/systemd/system/NetworkManager.service; disabled; vend>
     Active: inactive (dead)
       Docs: man:NetworkManager(8)
root@tegra-ubuntu:/home/denso# ifconfig eth0 up
root@tegra-ubuntu:/home/denso# dmesg | grep eth
[    0.000000] psci: probing for conduit method from DT.
[    2.863424] usbcore: registered new interface driver cdc_ether
[    4.453991] optee: probing for conduit method.
[    8.553929] nvethernet 2490000.ethernet: Adding to iommu group 30
[    8.563015] nvethernet 2490000.ethernet: failed to read skip mac reset flag, default 0
[    8.578455] nvethernet 2490000.ethernet: failed to read MDIO address
[    8.590126] nvethernet 2490000.ethernet: setting to default DMA bit mask
[    8.604906] nvethernet 2490000.ethernet: set default TXQ to TC mapping
[    8.623601] nvethernet 2490000.ethernet: Setting default PTP RX queue
[    8.629956] nvethernet 2490000.ethernet: Failed to read DMA Tx ring size, using default [1024]
[    8.638884] nvethernet 2490000.ethernet: Failed to read DMA Tx ring size, using default [1024]
[    8.652044] nvethernet 2490000.ethernet: failed to get eqos_rx_m clk
[    8.653571] nvethernet 2490000.ethernet: failed to get eqos_rx_input clk
[    8.664522] nvethernet 2490000.ethernet: failed to get eqos_tx_divider clk
[    8.665631] nvethernet 2490000.ethernet: Ethernet MAC address: 48:b0:2d:3c:83:66
[    8.696109] nvethernet 2490000.ethernet: Macsec not supported
[    8.703267] nvethernet 2490000.ethernet: eth1 (HW ver: 50) created with 1 DMA channels
[   12.250407] using random self ethernet address
[   12.250554] using random host ethernet address
[   13.670201] using random self ethernet address
[   13.670329] using random host ethernet address
[   53.603889] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
root@tegra-ubuntu:/home/denso# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 64512
        inet 10.10.10.3  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::78ba:9dff:fe8d:6634  prefixlen 64  scopeid 0x20<link>
        ether 7a:ba:9d:8d:66:34  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@tegra-ubuntu:/home/denso# ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2) 56(84) bytes of data.
From 10.10.10.3 icmp_seq=1 Destination Host Unreachable
From 10.10.10.3 icmp_seq=2 Destination Host Unreachable
From 10.10.10.3 icmp_seq=3 Destination Host Unreachable
From 10.10.10.3 icmp_seq=4 Destination Host Unreachable
From 10.10.10.3 icmp_seq=5 Destination Host Unreachable
From 10.10.10.3 icmp_seq=6 Destination Host Unreachable
From 10.10.10.3 icmp_seq=7 Destination Host Unreachable
From 10.10.10.3 icmp_seq=8 Destination Host Unreachable
From 10.10.10.3 icmp_seq=9 Destination Host Unreachable
From 10.10.10.3 icmp_seq=10 Destination Host Unreachable
From 10.10.10.3 icmp_seq=11 Destination Host Unreachable
From 10.10.10.3 icmp_seq=12 Destination Host Unreachable
From 10.10.10.3 icmp_seq=13 Destination Host Unreachable
^C
--- 10.10.10.2 ping statistics ---
17 packets transmitted, 0 received, +13 errors, 100% packet loss, time 16385ms
pipe 4
root@tegra-ubuntu:/home/denso# 

[WayneWWW]

Not only disabling NM, you need to set throught netplan as the topic

Jetpack5.0.2 Xavier pcie endpoint mode [repetition]

Hi @SooMSooM That patch is only needed for EP. I also tried on my devices and here are the setup. Both jp5.0.2 devices (rp/ep) can ping each other with this setup. this interface is easily to be affected by other network interface, so please try below Disable Network manager sudo systemctl stop NetworkManager.service sudo systemctl disable NetworkManager.service Set static IP use netplan.io to set static ip address → sudo apt install netplan.io → Static IP Address Assignment To c…

[hermes_wu]

Hi @WayneWWW ,

Yes, I’m set the static IP through Netplan already.

[hermes_wu]

Hello @WayneWWW ,

Do you have any other suggestions that I can verify this issue?