Hi,
I’ve set up one Xavier as a root, and a separate Xavier as an endpoint. I’m connecting the of them via iPass PCIe cards.
This is the endpoint card: https://www.onestopsystems.com/product/pcie-x8-gen3-dual-port-cable-adapter?no_cache=1610007541
This is the root card: EPCIE4XRDCA01 | PCIe Expansion | External PCI Express x4 (with Gen 1/2/3 Redriver) Cable Adapter Card | PCIe Cable Adapter | EPCIE4XRDCA01
My L4T version is 32.2.3.
The endpoint Xavier was set up the following way:
- Enabled bit 12 of ODMDATA in p2972-0000.conf.common (0x09190000 to 0x09191000)
- ./flash.sh jetson-xavier mmcblk0p1
cd /sys/kernel/config/pci_ep/
mkdir functions/pci_epf_nv_test/func1
echo 0x10de > functions/pci_epf_nv_test/func1/vendorid
echo 0x0001 > functions/pci_epf_nv_test/func1/deviceid
ln -s functions/pci_epf_nv_test/func1 controllers/141a0000.pcie_ep/
echo 1 > controllers/141a0000.pcie_ep/start
On the endpoint Xavier, I can use dmesg | grep pci_epf_nv_test
to get the RAM address, and read/write to it from the endpoint using busybox devmem.
However, the root Xavier doesn’t appear to have access to the endpoint Xavier. Running lspci -v
shows several PCIe bridges, rather than RAM memory: NVIDIA Corporation Device 0001
. Reading the memory at 0x1f40400000 always returns 0x874910B5, rather than memory set on the endpoint. Is there a way to gain visibility past these bridges?
Below is the host’s lspci -v
output:
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 34
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 40000000-400fffff
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [158] #26
Capabilities: [17c] #27
Capabilities: [190] L1 PM Substates
Capabilities: [1a0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [2a0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [2d8] #25
Capabilities: [2e4] Precision Time Measurement
Capabilities: [2f0] Vendor Specific Information: ID=0004 Rev=1 Len=054 <?>
Kernel driver in use: pcieport
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13) (prog-if 01 [AHCI 1.0])
Subsystem: Marvell Technology Group Ltd. Device 9171
Flags: bus master, fast devsel, latency 0, IRQ 563
I/O ports at 100010 [size=8]
I/O ports at 100020 [size=4]
I/O ports at 100018 [size=8]
I/O ports at 100024 [size=4]
I/O ports at 100000 [size=16]
Memory at 1230010000 (32-bit, non-prefetchable) [size=512]
Expansion ROM at 1230000000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Express Legacy Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: ahci
0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 38
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 00000000-00001fff
Memory behind bridge: 40000000-404fffff
Prefetchable memory behind bridge: 0000001c00000000-0000001c003fffff
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
Capabilities: [168] #26
Capabilities: [190] #27
Capabilities: [1c0] L1 PM Substates
Capabilities: [1d0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
Capabilities: [2d0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [308] #25
Capabilities: [314] Precision Time Measurement
Capabilities: [320] Vendor Specific Information: ID=0004 Rev=1 Len=054 <?>
Kernel driver in use: pcieport
0005:01:00.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 819
Memory at 1f40400000 (32-bit, non-prefetchable) [size=256K]
Bus: primary=01, secondary=02, subordinate=04, sec-latency=0
I/O behind bridge: 3a100000-3a101fff
Memory behind bridge: 40000000-403fffff
Prefetchable memory behind bridge: 0000001c00000000-0000001c003fffff
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Capabilities: [68] Express Upstream Port, MSI 00
Capabilities: [a4] Subsystem: One Stop Systems, Inc. Device 2302
Capabilities: [100] Device Serial Number 00-a0-d6-ff-ff-04-84-66
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [10c] #19
Capabilities: [148] Virtual Channel
Capabilities: [e00] #12
Capabilities: [b00] Latency Tolerance Reporting
Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcieport
0005:02:00.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 820
Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 3a100000-3a100fff
Memory behind bridge: 40000000-401fffff
Prefetchable memory behind bridge: 0000001c00000000-0000001c001fffff
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: One Stop Systems, Inc. Device 2302
Capabilities: [100] Device Serial Number 00-a0-d6-ff-ff-04-84-66
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [10c] #19
Capabilities: [148] Virtual Channel
Capabilities: [e00] #12
Capabilities: [f24] Access Control Services
Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcieport
0005:02:08.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 821
Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 3a101000-3a101fff
Memory behind bridge: 40200000-403fffff
Prefetchable memory behind bridge: 0000001c00200000-0000001c003fffff
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Capabilities: [68] Express Downstream Port (Slot+), MSI 00
Capabilities: [a4] Subsystem: One Stop Systems, Inc. Device 2302
Capabilities: [100] Device Serial Number 00-a0-d6-ff-ff-04-84-66
Capabilities: [fb4] Advanced Error Reporting
Capabilities: [138] Power Budgeting <?>
Capabilities: [10c] #19
Capabilities: [148] Virtual Channel
Capabilities: [e00] #12
Capabilities: [f24] Access Control Services
Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
Kernel driver in use: pcieport```