Ep BAR0_SIZE can not be set SZ_2G

In my pcie switcher system, there are one rc and two ep. Every ep’s BAR0_SIZE is SZ_512M. When i set BAR0_SIZE to SZ_2G, and on rc, lspci -v:
0005:12:00.0 RAM memory: NVIDIA Corporation Device 1ada
Flags: fast devsel, IRQ 39
Memory at 1c00000000 (64-bit, prefetchable) [disabled] [size=128K]
Memory at (64-bit, non-prefetchable) [disabled]

I download jetson4.6 source code.
how to support BAR0_SIZE to SZ_2G or SZ_4G?

How are you setting the size to 2G and do you observe any errors in the dmesg log?

1)pci-epf-nv-test.c中
ret = pci_epc_set_bar(epc, BAR_0, xdma->bar0_iova, BAR0_SIZE,
PCI_BASE_ADDRESS_SPACE_MEMORY |
PCI_BASE_ADDRESS_MEM_TYPE_32);

BAR0_SIZE定义为SZ_2G

2)rc上dmesg中log:
[ 5.689745] pci 0005:12:00.0: BAR 0: no space for [mem size 0x80000000]
[ 5.689748] pci 0005:12:00.0: BAR 0: failed to assign [mem size 0x80000000]

In pci-epf-nv-test.c,
when BAR0_SIZE=512M, lspci on rc device:


when BAR0_SIZE=SZ_2G, lspci on rc device:

When BAR0_SIZE=SZ_2G, dma_map_sg return error.

Hi William,
Would you please apply below and have a go ?
1.
diff --git a/drivers/pci/dwc/pcie-designware-ep.c b/drivers/pci/dwc/pcie-designware-ep.c
index 6424c63…b80b887 100644
— a/drivers/pci/dwc/pcie-designware-ep.c
+++ b/drivers/pci/dwc/pcie-designware-ep.c
@@ -169,6 +169,11 @@

dw_pcie_writel_dbi2(pci, reg, lower_32_bits(size - 1));
dw_pcie_writel_dbi(pci, reg, flags);

++
++ if (flags & PCI_BASE_ADDRESS_MEM_TYPE_64) {
++ dw_pcie_writel_dbi2(pci, reg + 4, upper_32_bits(size - 1));
++ dw_pcie_writel_dbi(pci, reg + 4, 0);
++ }
}

static int dw_pcie_ep_set_bar(struct pci_epc *epc, enum pci_barno bar,

  1. Also, Please don’t forget to modify the memory type form PCI_BASE_ADDRESS_MEM_TYPE_32 to PCI_BASE_ADDRESS_MEM_TYPE_64
    in drivers/pci/endpoint/functions/pci-epf-nv-test.c
    94 ret = pci_epc_set_bar(epc, BAR_0, epfnv->bar0_iova, BAR0_SIZE,
    95 PCI_BASE_ADDRESS_SPACE_MEMORY |
    96 PCI_BASE_ADDRESS_MEM_TYPE_64);

SZ_2G or SZ_4G does not exceed 32 bits, so why we need support PCI_BASE_ADDRESS_MEM_TYPE_64?

Good question! The reason is : not all low 4G memory can be allocated by PCIe.

Thanks, Jasonm!

I will try it later.

I try it. But rc device allocates space fail. The rc device’s dmesg in the file pcie_2G_error.txt.
pcie_2G_error.txt (91.9 KB)

Hi William,
Verified on my Xavier devkit.

  1. 2G seems ok:
    root@tegra-ubuntu:~# lspci -vvv -s 0005:01:00.0
    0005:01:00.0 RAM memory: NVIDIA Corporation Device 1ad5
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 819
    Region 0: Memory at 1f80000000 (64-bit, non-prefetchable) [size=2G]
    Region 2: Memory at 1c00000000 (64-bit, prefetchable) [size=128K]
    Region 4: Memory at 1f40000000 (64-bit, non-prefetchable) [size=1M]

  2. 4G failed.
    root@tegra-ubuntu:~# lspci -vvv -s 0005:01:00.0
    0005:01:00.0 RAM memory: NVIDIA Corporation Device 1ad5
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 819
    Region 0: Memory at “unassigned” (64-bit, prefetchable)

So, if really need to allocate 4G memory, maybe the “dma_alloc_coherent” no suitable.
An alternative:
a. reserved 4G memory
b. map the reserved memory to PCIe domain.

BR.,
Jason

Hi, Jasonm,

If rc device connects to ep device directly, 2G test is ok.
But one rc, one ep connected by switch, 2G test is fail.

Would’d you give me the file “hardware/nvidia/soc/t19x/kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi” please? I contrast with them.

tegra194-soc-pcie.dtsi (26.4 KB)
Hi William,
Attached.

Thanks.

If one rc, one or more ep connected by switch, Would you do the 2G test? I test this situation fail.

Hi William,
If one rc, one or more ep connected by switch

I would like to, but I didn’t have the pcie switch.
Only rc<–>ep directly.
By the way, would you please share your switch’s details?

BR.,
Jason

The pcie_2G_error.txt and pictures provided in previous discussion are my test results on switch.

Would you share your PCIe switch info like:
microsemi(Broadcom) /pcie 3.0 / port number…

Ok. I will upload it later.

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi william.fang,

Is this still an issue to support?

Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.