In this example I will show NVMe drives. However the problem exists with any PCIe device!
Orin does not properly set PCIe base address register 10h.
This register should point to the base address of Region 0.
When I query an Intel system it looks like this:
02:00.0 Non-Volatile memory controller: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD (rev 01) (prog-if 02 [NVM Express])
Subsystem: Sandisk Corp WD Black SN850
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
NUMA node: 0
IOMMU group: 14
Region 0: Memory at 75300000 (64-bit, non-prefetchable) [size=16K]
The PCIe base register correctly shows this address:
00: b7 15 11 50 07 04 10 00 01 02 08 01 10 00 00 00
10: **04 00 30 75** 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 b7 15 11 50
As you can see 753000 => 64 bit is correct.
Now when I investigate Orin:
root@orin:~# lspci -vv -s 0005:01:00.0 -xxx
0005:01:00.0 Non-Volatile memory controller: Toshiba Corporation Device 011a (prog-if 02 [NVM Express])
Subsystem: Toshiba Corporation Device 0001
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 68
Region 0: Memory at 2b28000000 (64-bit, non-prefetchable) [size=16K]
The address in the PCIe register is pointing to 0x40 00 00 => 64 bit.
00: 79 11 1a 01 06 04 10 00 00 02 08 01 00 00 00 00
10: 04 00 00 40 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 79 11 01 00
This is wrong, and no memory range has been registered at 0x40000000
Another problem is the PCIe endpoint driver.
It registers a 64 bit address as a 32 bit memory range, at least when the RC and Endpoint are both Orins.
To fix this something like this is needed or the Orin RC will see a 64 bit address registered as a 32-bit region.
diff -u pci-epf-nv-test.c-orig pci-epf-nv-test.c
--- pci-epf-nv-test.c-orig 2023-07-07 11:12:27.832075193 +0000
+++ pci-epf-nv-test.c 2023-08-09 07:31:41.869171369 +0000
@@ -47,7 +47,7 @@
epf_bar->size = BAR0_SIZE;
epf_bar->barno = BAR_0;
epf_bar->flags |= PCI_BASE_ADDRESS_SPACE_MEMORY |
- PCI_BASE_ADDRESS_MEM_TYPE_32;
+ PCI_BASE_ADDRESS_MEM_TYPE_64;
ret = pci_epc_set_bar(epc, epf->func_no, epf_bar);
if (ret) {
Thanks,
–Mark