Ep BAR0_SIZE can not be set SZ_2G

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

In my pcie switcher system, there are one rc and two ep. Every ep’s BAR0_SIZE is SZ_512M. When i set BAR0_SIZE to SZ_2G, and on rc, lspci -v：
0005:12:00.0 RAM memory: NVIDIA Corporation Device 1ada
Flags: fast devsel, IRQ 39
Memory at 1c00000000 (64-bit, prefetchable) [disabled] [size=128K]
Memory at (64-bit, non-prefetchable) [disabled]

I download jetson4.6 source code.
how to support BAR0_SIZE to SZ_2G or SZ_4G?

#2

How are you setting the size to 2G and do you observe any errors in the dmesg log?

#3

1）pci-epf-nv-test.c中
ret = pci_epc_set_bar(epc, BAR_0, xdma->bar0_iova, BAR0_SIZE,
PCI_BASE_ADDRESS_SPACE_MEMORY |
PCI_BASE_ADDRESS_MEM_TYPE_32);

BAR0_SIZE定义为SZ_2G

2）rc上dmesg中log:
[ 5.689745] pci 0005:12:00.0: BAR 0: no space for [mem size 0x80000000]
[ 5.689748] pci 0005:12:00.0: BAR 0: failed to assign [mem size 0x80000000]

#4

In pci-epf-nv-test.c，
when BAR0_SIZE=512M, lspci on rc device:

image
image1422×1042 282 KB

when BAR0_SIZE=SZ_2G, lspci on rc device:
image
image1400×948 244 KB

#5

When BAR0_SIZE=SZ_2G, dma_map_sg return error.

#6

Hi William,
Would you please apply below and have a go ?
1.
diff --git a/drivers/pci/dwc/pcie-designware-ep.c b/drivers/pci/dwc/pcie-designware-ep.c
index 6424c63…b80b887 100644
— a/drivers/pci/dwc/pcie-designware-ep.c
+++ b/drivers/pci/dwc/pcie-designware-ep.c
@@ -169,6 +169,11 @@

dw_pcie_writel_dbi2(pci, reg, lower_32_bits(size - 1));
dw_pcie_writel_dbi(pci, reg, flags);

++
++ if (flags & PCI_BASE_ADDRESS_MEM_TYPE_64) {
++ dw_pcie_writel_dbi2(pci, reg + 4, upper_32_bits(size - 1));
++ dw_pcie_writel_dbi(pci, reg + 4, 0);
++ }
}

static int dw_pcie_ep_set_bar(struct pci_epc *epc, enum pci_barno bar,

  1. Also, Please don’t forget to modify the memory type form PCI_BASE_ADDRESS_MEM_TYPE_32 to PCI_BASE_ADDRESS_MEM_TYPE_64
    in drivers/pci/endpoint/functions/pci-epf-nv-test.c
    94 ret = pci_epc_set_bar(epc, BAR_0, epfnv->bar0_iova, BAR0_SIZE,
    95 PCI_BASE_ADDRESS_SPACE_MEMORY |
    96 PCI_BASE_ADDRESS_MEM_TYPE_64);
#7

SZ_2G or SZ_4G does not exceed 32 bits, so why we need support PCI_BASE_ADDRESS_MEM_TYPE_64?

#8

Good question! The reason is : not all low 4G memory can be allocated by PCIe.

#9

Thanks, Jasonm!

I will try it later.

#10

I try it. But rc device allocates space fail. The rc device’s dmesg in the file pcie_2G_error.txt.
pcie_2G_error.txt (91.9 KB)

#11

Hi William,
Verified on my Xavier devkit.

  1. 2G seems ok:
    root@tegra-ubuntu:~# lspci -vvv -s 0005:01:00.0
    0005:01:00.0 RAM memory: NVIDIA Corporation Device 1ad5
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 819
    Region 0: Memory at 1f80000000 (64-bit, non-prefetchable) [size=2G]
    Region 2: Memory at 1c00000000 (64-bit, prefetchable) [size=128K]
    Region 4: Memory at 1f40000000 (64-bit, non-prefetchable) [size=1M]

  2. 4G failed.
    root@tegra-ubuntu:~# lspci -vvv -s 0005:01:00.0
    0005:01:00.0 RAM memory: NVIDIA Corporation Device 1ad5
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 819
    Region 0: Memory at “unassigned” (64-bit, prefetchable)

So, if really need to allocate 4G memory, maybe the “dma_alloc_coherent” no suitable.
An alternative:
a. reserved 4G memory
b. map the reserved memory to PCIe domain.

BR.,
Jason

#12

Hi, Jasonm,

If rc device connects to ep device directly, 2G test is ok.
But one rc, one ep connected by switch, 2G test is fail.

Would’d you give me the file “hardware/nvidia/soc/t19x/kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi” please? I contrast with them.

#13

tegra194-soc-pcie.dtsi (26.4 KB)
Hi William,
Attached.

#14

Thanks.

If one rc, one or more ep connected by switch, Would you do the 2G test? I test this situation fail.

#15

Hi William,
If one rc, one or more ep connected by switch

I would like to, but I didn’t have the pcie switch.
Only rc<–>ep directly.
By the way, would you please share your switch’s details?

BR.,
Jason

#16

The pcie_2G_error.txt and pictures provided in previous discussion are my test results on switch.

#17

Would you share your PCIe switch info like:
microsemi(Broadcom) /pcie 3.0 / port number…

#18

Ok. I will upload it later.

#19

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi william.fang,

Is this still an issue to support?

Thanks

#20

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.