How to map kernel memory to user memory?

 Now I developed a PCIE-card driver,The dma_proxy kernel module works correctly and performs the INTERNAL_TEST correctly, but when I try to read/write on the dma buffers after mapping them on the user application i get this error:

Unhandled fault: level 3 address size fault (0x92000043) at 0x0000007f7acc101c.

 But I can pass same code on a AMD64 machine!

 I use following code to realize:

 gReadBuffer = pci_alloc_consistent(gDev, BUF_SIZE, &gReadHWAddr); 
 gBasePhy=virt_to_phys((void *)gReadBuffer);

 static int XPCIe_mmap(struct file *filp, struct vm_area_struct *vma)
 {
unsigned long size = vma->vm_end - vma->vm_start;    
remap_pfn_range(vma,vma->vm_start,gBasePhy>> PAGE_SHIFT,size,vma->vm_page_prot);
vma->vm_flags |= ( VM_IO | VM_DONTEXPAND | VM_DONTDUMP );
    vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
    printk("start %lx end %lx off %lx \n",vma->vm_start,vma->vm_end,vma->vm_pgoff);
return 0;
 }

 How to resolve the problem?

This answer I find form forum:https://devtalk.nvidia.com/default/topic/1002486/?comment=5137355. But how to solve it on TX2 platform.

Can I use /dev/mem to access the phys Memory?

/dev/mem can be used to access physical memory, but, when SMMU is enabled for PCIe, bus address that PCIe end points are given to access host system memory is not equivalent to physical memory. For time being, you can disable SMMU to have your code working.

Thank you very much! But how to disable SMMU on TX2 platform?

Hello,

I have issue with Intel’s igb driver when SMMU is anabled for PCIE controller. There was a simple solution how to disable SMMU on TX1 by removing a line in one of device tree spec file. Could you please provide a solution for TX2 also? I am not very familiar with device tree specifications.

Thanks

Yes,this link tell us how to disable SMMU on TX1, but how about TX2?
https://devtalk.nvidia.com/default/topic/988351/jetson-tx1/what-is-the-current-status-of-pcie-dma-/1

Hi,
I have the same issue, also on a tx2
removing the line as stated in the link does not solve the issue

Is there an easy way to disable SMMU on TX2?

following need to be removed.

  1. #stream-id-cells = <1>;” from “tegra_pcie” node
  2. “<&{/pcie-controller@10003000} TEGRA_SID_AFI>,” from “smmu” node

Where is the full device tree file? I can’t find these line you mentioned!

Hi,

Some grep’ing found a file named tegra186-soc-base.dtsi that contains the lines, but even after commenting them out i still get level 3 address size fault errors

You may want to verify that the device tree you expect is really being used (kernel has a “make dtbs” target for building device tree…else it may not have built changes). In “/boot/extlinux/extlinux.conf” you will see a DTC entry naming the dtb file actually used (U-Boot itself may use a different dtb, but the named dtb is what the kernel is given at the moment it loads). You can extract or re-compile this dtb file…here is an example:

dtc -I dtb -O dts -o /tmp/extracted.dts /boot/the_firmware_in_extlinux.dtb
dtc -I dts -O dtb -o /tmp/modified.dtb /tmp/extracted.dts

Do note that the kernel version may contain comments, and comments are lost when compiled. Some variables may also use more meaningful names within the kernel code, but mostly it is easy to see what belongs where in a reverse compile.

@kabraham , after making changes to DTB and reflashing, can you confirm if SMMU is disabled for PCIe? Simple way to check is @ /sys/kernel/debug/12000000.iommu/masters/ path, there shouldn’t be any PCIe entries if SMMU is disabled for PCIe.

Hi,

this is the contents of that folder, i don’t see any pcie entries:

0000:00:01.0/ 13e10000.host1x:ctx6/ 15810000.se/ 3180000.i2c/ 3550000.xudc/
0000:01:00.0/ 13e10000.host1x:ctx7/ 15820000.se/ 3190000.i2c/ adsp_audio/
0000:02:08.0/ 150c0000.nvcsi/ 15830000.se/ 31b0000.i2c/ b000000.rtcpu/
0000:03:00.0/ 15100000.tsecb/ 15840000.se/ 31c0000.i2c/ c160000.aon/
10003000.pcie-controller/ 15210000.nvdisplay/ 17000000.gp10b/ 31e0000.i2c/ c240000.i2c/
13e10000.host1x/ 15340000.vic/ 2490000.ether_qos/ 3210000.spi/ c250000.i2c/
13e10000.host1x:ctx0/ 15380000.nvjpg/ 2600000.dma/ 3240000.spi/ c260000.spi/
13e10000.host1x:ctx1/ 15480000.nvdec/ 2993000.adsp/ 3400000.sdhci/ d000000.bpmp/
13e10000.host1x:ctx2/ 154c0000.nvenc/ 3100000.serial/ 3460000.sdhci/ smmu_test/
13e10000.host1x:ctx3/ 15500000.tsec/ 3110000.serial/ 3507000.ahci-sata/ sound/
13e10000.host1x:ctx4/ 15600000.isp/ 3130000.serial/ 3510000.hda/
13e10000.host1x:ctx5/ 15700000.vi/ 3160000.i2c/ 3530000.xhci/

If it’s not the SMMU, it could possibly be something with the arm architechture, as the driver works on x86. I know there are some differences in allignment handling, however, i would expect a different error when that happens.
Are there any other differences in pcie device handling? Or could there still be some remainders of the SMMU?

Thanks,

kabraham

Edit: just saw 10003000.pcie-controller/, so maybe something went wrong… I’ll try make dtbs as linuxdev suggested

I still see PCIe entries… so… as you guessed correctly, something went wrong and your DT changes didn’t reflect on target.

Hi,

I tried a make dtbs, it clearly builds the .dtbs file (i tryed adding some invalid syntax) but still even after a new clean build after copying the image I still have the pcie-controller entry, and the module fails.

I didn’t see any DTC entry in /boot/extlinux/extlinux.conf, and i don’t know where any other image should come from, but i’ll have a look again tomorrow morning

Appreciate the help,
kabraham

If this is the developer board and there is no DTC entry (the FDT key/value pair) in extlinux.conf you could add one. Building the dtb file in the kernel instead of editing one which is in place would still require a FDT entry…the only difference is if the file came from device tree compiler or from reverse-compile/edit/recompile.

As an example, I added an entry for enabling a serial port. I started with an existing dtb file, “tegra186-quill-p3310-1000-c03-00-base.dtb”, which you will find matches the running kernel (U-Boot has set up a device tree which basically matches this…which is why it works to not have the dtb file). I edited this through reverse-compile, then recompiled with the “modified_” name prefixing the original:

LABEL ths2
      MENU LABEL ths2
      <b>FDT /boot/modified_tegra186-quill-p3310-1000-c03-00-base.dtb</b>
      LINUX /boot/Image
      APPEND fbcon=map:0 net.ifnames=0 console=tty0 OS=l4t console=ttyS0,115200n8 memtype=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x03100000 gpt tegraid=18.1.2.0.0 tegra_keep_boot_clocks maxcpus=6 android.kerneltype=normal androidboot.serialno=0335115020673 vpr_resize root=/dev/mmcblk0p1 rw rootwait

I would suggest to reverse compile /boot/tegra186-quill-p3310-1000-c03-00-base.dtb, edit the part you want, recompile it, and put in the modified name as I did as an FDT key/value pair entry.

I do the experiment which @linuxdev said! But I still receive the Level 3 error Message!
My /sys/kernel/debug/12000000.iommu/masters path’s contents like below:

13e10000.host1x 13e10000.host1x:ctx6 15480000.nvdec 15830000.se 3110000.serial 31e0000.i2c 3530000.xhci c260000.spi
13e10000.host1x:ctx0 13e10000.host1x:ctx7 154c0000.nvenc 15840000.se 3130000.serial 3210000.spi 3550000.xudc d000000.bpmp
13e10000.host1x:ctx1 150c0000.nvcsi 15500000.tsec 17000000.gp10b 3160000.i2c 3240000.spi adsp_audio smmu_test
13e10000.host1x:ctx2 15100000.tsecb 15600000.isp 2490000.ether_qos 3180000.i2c 3400000.sdhci b000000.rtcpu sound
13e10000.host1x:ctx3 15210000.nvdisplay 15700000.vi 2600000.dma 3190000.i2c 3460000.sdhci c160000.aon
13e10000.host1x:ctx4 15340000.vic 15810000.se 2993000.adsp 31b0000.i2c 3507000.ahci-sata c240000.i2c
13e10000.host1x:ctx5 15380000.nvjpg 15820000.se 3100000.serial 31c0000.i2c 3510000.hda c250000.i2c

which seems right but I’m always Wrong!
BTW I still get the same address form virt_to_phys() fun. Maybe there the SMMU still effect?

Did you use a serial console to boot to the alternate boot entry? Or did you add the FDT to the existing extlinux.conf default entry? FYI, you can see a reflection of what is in the device tree by examining “/proc/device-tree/”…your edits should show up as matching the file system tree within “/proc/device-tree/”…so you can at least verify if your device tree was used or if for some reason it was ignored.

@linuxdev: I’m sure I use the changed DTB. Because the “pcie” line was disappeared when I comment these line which @vidyas said! Now I always get the same error , the virt_to_phys() fun return same phy address like before,the address always FFFFFFC080E81000. I think these change cann’t get the real phy address!