DMA from PCIe Device to Jetson Tx2 local DDR

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

Hi ,

I am using FPGA(Xilinx) board as PCIe Device (EP) in my project , and using Jetson Tx2 as host side (RC). We excute the PCIE insmod file on Tx2 and now we can read and write to the FPGA successfully using PIO mode . However when the jetson starts FPGA DMA to local ddr , jetson tx2 report an error . I am sure the DMA from FPGA is correct with testing on another host PC .
So what is the problem ?
Does Jetson Tx2 hardware support DMA from PCIe Device to Jetson Tx2 local DDR ?

Thanks.

#2

Does Jetson Tx2 hardware support DMA from PCIe Device to Jetson Tx2 local DDR ?
Yes. It does support. Can you please give more info on the error you are getting?

#3

Hi vidays,

The DMA problem has been solved by update the system to r28.1, thank you!
But I encounter another issue now. We use the PCIe transmitting 25MB datas to the DDR and use “mmap function” mapping the DDR (kernel space) to user space, like this

In probe function:
pdev->buf_vir_addr = dma_alloc_coherent(pci_dev, 25 * 1024 * 1024, &pdev->buf_dma_addr, GFP_KERNEL);
memset(pdev->buf_vir_addr, 0xAA, 25 * 1024 * 1024);

In mmap function:

int mydev_mmap(struct file *filp, struct vm_area_struct *vma)
{
int ret = 0;
vma->vm_flags |= (VM_IO | VM_DONTEXPAND | VM_DONTDUMP);
ret = remap_pfn_range(vma, vma->start, pdev->buf_dma_addr >> PAGE_SHIFT, vma->end - vma->start, PAGE_SHARED);

return ret;
}

The problem is that we can not get the right value (0xAA) in the user space. After we change the “dma_alloc_coherent(pci_dev, 25 * 1024 * 1024, &pdev->buf_dma_addr, GFP_KERNEL)” to "dma_alloc_coherent(NULL, 25 * 1024 * 1024, &pdev->buf_dma_addr, GFP_KERNEL), the mmap function work fine. But the PCIe can not work again.

When use pci_dev as the parameter, the buf_dma_addr is start at 0x80000000; use NULL, the buf_dma_addr is start at 0xFx000000.

How to solve the contradiction?

Thanks.

#4

Hi,
Your observations are all expected.
Since SMMU is enabled for PCIe, using bus address as an input to remap_pfn_range() will not work.
Also, passing a null to dma_alloc_coherent() wouldn’t create mapping taking PCIe into account.
You may have to disable SMMU for PCIe.
Also, you might want to take a look at https://devtalk.nvidia.com/default/topic/1014290/jetson-tx2/how-to-map-kernel-memory-to-user-memory-/ once.

#5

Also, you can try using iommu_iova_to_phys() API to get physical address of iova and then pass it on to remap_pfn_range() API.

#6

Hi vidays,

According to your suggestion disabled the PCIe SMMU, it looks like working fine. And i want to know which DTB file is the default one for TX2? Because I use the “tegra186-quill-p3310-1000-c03-00-base.dtb” as booting DTB file, the wired connection can not work.

Thank you very much! I will try iommu_iova_to_phys() API later.

#7

I have try to use iommu_iova_to_phys() and remap_pfn_range() APIs. But it can map 4MB space only.

#8

that is probably because physical address corresponding to iova may not be contiguous and in this particular case, only 4MB (starting from the address returned by iommu_iova_to_phys() API) is contiguous.

#9

Hi!
I use L4T 28.2 with modification of the CMA size (CONFIG_CMA_SIZE_MBYTES parameter) set to 128.
In the the dtb file tegra186-quill-p3310-1000-c03-00-base.dtb there is no line for disabling SMMU as multiples topics suggest to do.
All the application code have been tested with success on a x86 platform, but at present moment on the jetson board, mmap function (with dma_mmap_coherent) do not give the good virtual address for the buffer allocate in my kernel PCIe module (with dma_alloc_coherent).
How can I disable SMMU for the pcie controler?