I am working on a project where FPGA has PCIe endpoint connected to PCIe Root of TX1 and act as bus master meaning that the FPGA can initiate data transfer upstream to TX1. I configured my FPGA AXI_to_PCIe BAR for some random memory address (0x2000_0000) but the truth is that, being an FPGA guy, I have no idea how and where to map the PCIe BAR address. Do I need to reserve an address space for thisand just hard-code it into the FPGA PCIe? We need to DMA-copy data from FPGA to TX1 memory for GPU to be able to do some processing with it later. Can you please advice?
I am able to lspci and correctly configure the FPGA PCIe endpoint to be bus master and enable it.
I’m not sure if I understood your requirement correctly, but, let me summarize what I understood.
“You have an FPGA based PCIe end point and you want to DMA some data from FPGA based PCIe end point’s internal memory to TX1’s system memory”
If this is what your requirement is, all you need is an internal memory in FPGA based end point, which doesn’t have to be exposed to system through BAR window (as you anyway want to use EP’s internal DMA engine to DMA it to TX1 system memory). Exposing that memory through BAR comes into picture if there is no internal DMA in FPGA end point and you want system (i.e. TX1’s CPU) to read/write to your FPGA’s internal memory through BAR.
You might still want to have a BAR window exposed by your EP with some control registers which, when programmed by TX1, would trigger EP’s internal DMA engine to start read/write operations to TX1’s system memory.
Thanks for taking time to answer. I agree with what you said and that’s what I have in my system. My real question is how to let the FPGA EP to know the destination address of the TX1 memory to copy the DMA transfer to it. Yes the FPGA has a DMA that copies some data from FPGA memory to TX1 memory trough the PCIe bridge. Linux knows the destination address where it wants the FPGA EP to copy data. So how to let the FPGA know? Is there a Linux function that will set some configuration in the config space of the FPGA EP for the destination address? Or (and as I am now doing) TX1 has access to some registers in the FPGA through a PICe BAR and it uses that to write/configure that destination address to. The FPGA logic reads that register and assign the DMA start address with that read value…
Do you have a PCIe end point driver in your setup running on host linux? If yes, that is where we generally allocate memory in system through standard calls like dma_alloc_coherent() or dma_map_single() based on requirement, and then pass on that bus address to your FPGA-EP. Well, you asked us how to do that, but that truely depends on your end point design. In general, end points either have a register (to be written by host system with this address information) as part of their BAR (mostly) or config space.
If that register is part of BAR, claim the BAR (again, all this is pretty much standard and any PCIe device driver code [Ex: r8169.c] would show you how to do that) and access it through writel() call.
If that register is part of config space, you can use pci_write_config_word() standard API to write to that register specifying the offset.