Now I developed a PCIE-card driver,The dma_proxy kernel module works correctly and performs the INTERNAL_TEST correctly, but when I try to read/write on the dma buffers after mapping them on the user application i get this error:
Unhandled fault: level 3 address size fault (0x92000043) at 0x0000007f7acc101c.
But I can pass same code on a AMD64 machine!
I use following code to realize:
gReadBuffer = pci_alloc_consistent(gDev, BUF_SIZE, &gReadHWAddr);
gBasePhy=virt_to_phys((void *)gReadBuffer);
static int XPCIe_mmap(struct file *filp, struct vm_area_struct *vma)
{
unsigned long size = vma->vm_end - vma->vm_start;
remap_pfn_range(vma,vma->vm_start,gBasePhy>> PAGE_SHIFT,size,vma->vm_page_prot);
vma->vm_flags |= ( VM_IO | VM_DONTEXPAND | VM_DONTDUMP );
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
printk("start %lx end %lx off %lx \n",vma->vm_start,vma->vm_end,vma->vm_pgoff);
return 0;
}
How to resolve the problem?
/dev/mem can be used to access physical memory, but, when SMMU is enabled for PCIe, bus address that PCIe end points are given to access host system memory is not equivalent to physical memory. For time being, you can disable SMMU to have your code working.
I have issue with Intel’s igb driver when SMMU is anabled for PCIE controller. There was a simple solution how to disable SMMU on TX1 by removing a line in one of device tree spec file. Could you please provide a solution for TX2 also? I am not very familiar with device tree specifications.
Some grep’ing found a file named tegra186-soc-base.dtsi that contains the lines, but even after commenting them out i still get level 3 address size fault errors
You may want to verify that the device tree you expect is really being used (kernel has a “make dtbs” target for building device tree…else it may not have built changes). In “/boot/extlinux/extlinux.conf” you will see a DTC entry naming the dtb file actually used (U-Boot itself may use a different dtb, but the named dtb is what the kernel is given at the moment it loads). You can extract or re-compile this dtb file…here is an example:
Do note that the kernel version may contain comments, and comments are lost when compiled. Some variables may also use more meaningful names within the kernel code, but mostly it is easy to see what belongs where in a reverse compile.
@kabraham , after making changes to DTB and reflashing, can you confirm if SMMU is disabled for PCIe? Simple way to check is @ /sys/kernel/debug/12000000.iommu/masters/ path, there shouldn’t be any PCIe entries if SMMU is disabled for PCIe.
If it’s not the SMMU, it could possibly be something with the arm architechture, as the driver works on x86. I know there are some differences in allignment handling, however, i would expect a different error when that happens.
Are there any other differences in pcie device handling? Or could there still be some remainders of the SMMU?
Thanks,
kabraham
Edit: just saw 10003000.pcie-controller/, so maybe something went wrong… I’ll try make dtbs as linuxdev suggested
I tried a make dtbs, it clearly builds the .dtbs file (i tryed adding some invalid syntax) but still even after a new clean build after copying the image I still have the pcie-controller entry, and the module fails.
I didn’t see any DTC entry in /boot/extlinux/extlinux.conf, and i don’t know where any other image should come from, but i’ll have a look again tomorrow morning
If this is the developer board and there is no DTC entry (the FDT key/value pair) in extlinux.conf you could add one. Building the dtb file in the kernel instead of editing one which is in place would still require a FDT entry…the only difference is if the file came from device tree compiler or from reverse-compile/edit/recompile.
As an example, I added an entry for enabling a serial port. I started with an existing dtb file, “tegra186-quill-p3310-1000-c03-00-base.dtb”, which you will find matches the running kernel (U-Boot has set up a device tree which basically matches this…which is why it works to not have the dtb file). I edited this through reverse-compile, then recompiled with the “modified_” name prefixing the original:
I would suggest to reverse compile /boot/tegra186-quill-p3310-1000-c03-00-base.dtb, edit the part you want, recompile it, and put in the modified name as I did as an FDT key/value pair entry.
I do the experiment which @linuxdev said! But I still receive the Level 3 error Message!
My /sys/kernel/debug/12000000.iommu/masters path’s contents like below:
Did you use a serial console to boot to the alternate boot entry? Or did you add the FDT to the existing extlinux.conf default entry? FYI, you can see a reflection of what is in the device tree by examining “/proc/device-tree/”…your edits should show up as matching the file system tree within “/proc/device-tree/”…so you can at least verify if your device tree was used or if for some reason it was ignored.
@linuxdev: I’m sure I use the changed DTB. Because the “pcie” line was disappeared when I comment these line which @vidyas said! Now I always get the same error , the virt_to_phys() fun return same phy address like before,the address always FFFFFFC080E81000. I think these change cann’t get the real phy address!