Help for writing a dma driver for data transfert betwen two xavier connected by PCIe

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hi,

I have two Xavier AGX connected with a PCIe link. One of them was flashed as Root-Complex.
Currently, I am using the “Virtual Ethernet over PCIe” driver and C ++ sockets for data transfers between my two Xavier. However, the current bandwidth of the “Virtual Ethernet over PCIe” driver is around 5 GBits / s, which is insufficient for the needs of our application.

If I understood correctly :

  • The “pcie-tegra-dw-ep.c” file is the platform driver (ie the “PCIe EndPoint controller driver”)
  • The “tegra-pcie-ep-mem.c” file is the client driver. Is this the “PCie EndPoint function driver”? It is this driver that manages DMA operations, from what I understand.
  • The “pci-epf-nv-test.c” file is the driver used to test the “RAM memory” function of associated EndPoint controller.

When starting the system, neither lsmod nor dmesg gives anything on any of these three drivers. After executing the instructions given in the section “Connecting and configuring the systems” , I find messages concerning the driver “pci-epf-nv-test” and the phrase “EP init done” (coming from the driver “tegra-pci-dw” for the 141a0000.pcie_ep controller) with dmesg. However, lsmod still does not display them.
Are the “pcie-tegra-dw-ep” and “tegra_ep_mem” drivers loaded? If so, when and how can it be verified?

I am very new to DMA and PCIe drivers. So, I apologize in advance if my questions seem silly.

I browsed the “tegra-pcie-ep-mem.c” file:

  • Line 807, remap 1MB for Region 4. Verifiable with lspci. As I understand it, these in these 1MB, are the DMA configuration registers mapped?
  • Line 817, allocate 10MB. Not visible with lspci. What are these 10MB used for? How to use them in an application use?

On the Xavier in Root-Complex mode, we can see on the output of lspci:

xavier-rc @ xavierrc-desktop: ~ $ sudo lspci -s 0005: 04: 00.0 -vv
0005: 04: 00.0 RAM memory: NVIDIA Corporation Device 0001
Control: I / O- Mem + BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap + 66MHz- UDF- FastB2B- ParErr- DEVSEL = fast> TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 255
Region 0: Memory at 1f40200000 (32-bit, non-prefetchable) [size = 64K]
Region 2: Memory at 1c00000000 (64-bit, prefetchable) [size = 128K]
Region 4: Memory at 1f40100000 (64-bit, non-prefetchable) [size = 1M]

I understand that the 64KB is allocated in by the driver “pci-epf-nv-test”. And, does that correspond to the space of the PCIe device configuration registers?
The 128KB are allocated by which driver? What is this region used for?

If I want to be able to have 2GB of memory for my DMA transfers, which part of which driver should I modify?

Would there be a document describing the DMA registers (corresponding to the C5 controller)?

Must I load the driver “tegra_ep_mem” on my Root-Complex?

On my Xavier that operate as EP, instead of using busybox, how can I send data to my Root-Complex, as well as read data received from it?

Thank you in advance for your answers.

2

There may be other Linux forums that may be better to answer this question. The Xavier is running Ubuntu Linux and your question on drivers really is not Linux-for-Tegra specific.

Did you have to rebuilt the kernel to add the pci driver and add it to the .config file and a LKM (m) or built in driver (y)?