PCIE SGDMA Problem between xavier & FPGA

Hi there
Recently I’ve been working on “xavier + fpga” platform, where FPGA are connected through the PCIE Gen2 X4 bus.
‘lspci -vv’
we observed our FPGA (a pcie endpoint device),

0005:01:00.0 Unassigned class [ff80]: Xilinx Corporation Device 7024
Subsystem: Xilinx Corporation Device 0007
… …

this implies the FPGA has been recognized by the xavier through the PCIE Gen2 x4 bus. But during the 'Driver Development ’ I have found a problem on PCIE Master Write.

FPGA (the DMA MAster) actively writes to the xavier.

Thw workflow is as follows:

  1. In the driver we allocated consistent buffer for SGDMA descriptor and data buffer which to receive FPGA data:
    descriptor_vir = pci_alloc_consistent(gDev, 4096, &descriptor_phy);
    xavier_vir = pci_alloc_consistent(gDev, 64, &xavier_phy);
    when I printk these 2 addresses:
    descriptor_vir ffffffc38de3e000, descriptor_phy ffffe000
    xavier_vir ffffffc38dd7f000, xavier_phy ffffd000

  2. descriptor include xavier_phy;

  3. Pass the ‘descriptor_phy’ (as u64) to the FPGA through PCI BAR0 memory region.

  4. Then start the Master DMA Write.

  5. Read Interrupt status, and got nothing.

I’ve tested there process on PC, it worked fine. I thought this may be the bus address issue.

FPGA writes consecutive numbers:

0x0, 0x1, 0x2, 0x3, …, 0x1F,
0x0, 0x1, 0x2, 0x3, …, 0x1F,

To ‘xavier_phy’
upon the completion of DMA Master Write, we performed a check over the allocated DMA buffer, i.e.,

((u64*)virAddr)[0],
((u64*)virAddr)[1],
((u64*)virAddr)[2],

They turned out to be all 0…

I’ve checked https://devtalk.nvidia.com/default/topic/1026334/jetson-tx2/pcie-dma-problem-between-tx2-amp-fpga/post/5220805/#5220805

Any help, please!

descriptor_vir = pci_alloc_consistent(gDev, 4096, &descriptor_phy);
xavier_vir = pci_alloc_consistent(gDev, 64, &xavier_phy);
when I printk these 2 addresses:
descriptor_vir ffffffc38de3e000, descriptor_phy ffffe000
xavier_vir ffffffc38dd7f000, xavier_phy ffffd000

Are those addresses right?

Ideally, this should have worked fine (as it works fine for all the off-the-shelf PCIe cards with upstreamed drivers on Jetson-Xavier).
BTW, I have few points here…
How are you getting the ‘gDev’ here?
Can you please try with the API dma_alloc_coherent()?