I am unsure of how the physical/virtual translation works on PCIe DMA. Now that you can confirm device tree though I suspect someone will be able to offer the answer.
There should be a way to get the pages even with the smmu enabled. I think that would be a much nicer solution
After I disabled ASPM for PCIe it started working, but I had to disable it through /proc fs, not using pcie_aspm=off kernel boot parameter. Here is a complete list of steps I did:
- disable SMMU according vydias’ post
- add FDT entry into /boot/extlinux/extlinux.conf like linuxdev wrote
- disable ASPM:
sh$ echo performance | tee /sys/module/pcie_aspm/parameters/policy
I’ve tested it couple of days and everything works according to expectations.
Anyway, I agree with MartijnBerger.
Are you sure you are working correctly? I do the experiment u said and I get the same error! I mapped kernel memory which allocated by pci_alloc_consistent(), but the virt_to_phys() fun return the same phy address FFFFFFC080E81000. Are you use virt_to_phys() function and mmap() function?
One thing I forgot to mention is that I patched kernel according to patch posted by vydias here:
Regarding the addresses, I didn’t check addresses returned by virt_to_phys(), I only checked data contained in mmap()-ed area, which is filled in by kernel and read in userspace. And those are correct. I’ll try to find time for checking the addresses and let you know during this week.
Everyone:
Thank you for ur help! I get the right result now. @dusancerhaty's suggest is very help full but ASPM off is not necessary in my driver. The step is like below:
1. patched kernel for DMA :<a target='_blank' rel='noopener noreferrer' href='https://devtalk.nvidia.com/default/topic/1002486/jetson-tx2/iommu-unhandled-context-fault-on-pci-device-dma/post/5127145/#5127145'>https://devtalk.nvidia.com/default/topic/1002486/jetson-tx2/iommu-unhandled-context-fault-on-pci-device-dma/post/5127145/#5127145</a>
2. disable SMMU according vydias' post
3. add FDT entry into /boot/extlinux/extlinux.conf like linuxdev wrote
4. Write right mmap() fun in ur drvier.
My driver’s detail like below:
gReadBuffer = pci_alloc_consistent(gDev, BUF_SIZE, &gReadHWAddr);
static int XPCIe_mmap(struct file *filp, struct vm_area_struct *vma)
{
unsigned long size = vma->vm_end - vma->vm_start;
vma->vm_flags |= ( VM_IO | VM_DONTEXPAND | VM_DONTDUMP );
if(remap_pfn_range(vma,vma->vm_start,<b>vmalloc_to_pfn(gReadBuffer)</b>,size,vma->vm_page_prot))
printk("vmalloc_to_pfn Error! \n");
return 0;
}
Note: The bold line is very important for memory which allocated by pci_alloc_consistent() fun! don’t use
remap_pfn_range(vma,vma->vm_start,gBasePhy>> PAGE_SHIFT,size,vma->vm_page_prot);
Because that used for Kmalloc() fun.
Hello,
Can confirm that this works, thank you everyone!
These steps work fine. Still looking for a solution to do this with the smmu enalabled
also it seems that L4T 28.1 's kernel is branched and that the DMA32 issue is still in drivers/pci/host/pci-tegra.c
@vidyas from looking at the tag the patch mentioned in this thread:
- msi->pages = __get_free_pages(GFP_DMA32, 0);
+ msi->pages = __get_free_pages(GFP_DMA, 0);
was not applied. I have not verified if this is a problem or just looks like one.
I need to port my changes to the new kernel, ill let you know next week
Hi everyone,
I alse meet this problem,but I am not familier with driver programming.
I work with a platform of FPGA + TX2.I need to get data from FPGA to TX2 with PCIE.
Originally, I find a physical address large enough then I give this address to FPGA.This work normally in TX1,but when I move to TX2 it doesn’t work my TX2 release is R28.1.
I try to disabble SMMU using the following method：
remove：
- “#stream-id-cells = <1>;” from “tegra_pcie” node
- “<&{/pcie-controller@10003000} TEGRA_SID_AFI>,” from “smmu” node
But my TX2 still can not work.
I also add FDT entry into /boot/extlinux/extlinux.conf and patched kernel for DMA.
Unfortunately,it still can not work.
I am eager to know the answer,
Thanks.
The way of applying device tree DB in L4T version R28.1 has changed. You have to put your self-compiled *.dtb files into <YOUR_JETPACKDIR>/64_TX2/Linux_for_Tegra_tx2/kernel/dtb and then run:
$$ sudo ./flash.sh -k kernel-dtb jetson-tx2 mmcblk0p1
while your devkit is in recovery mode. For more info look at:
[url]TX1 can't boot with no HDMI in R28.1 - Jetson TX1 - NVIDIA Developer Forums
I intentionaly removed -r option, 'cause it’s obsolete and it’ll end up in failure, in case you use it.
Hello chay1991
I have same problem to disable SMMU on TX2 board R28.1
I dumped the dts file from the current running dtb file on the board using command:
dtc –I dtb –O dts -o /tmp/extracted_from_dtb.dts /boot/tegra186-quill-p3310-1000-c03-00-base.dtb
To disable SMMU . we need to do:
remove：
- “#stream-id-cells = <1>;” from “tegra_pcie” node
- “<&{/pcie-controller@10003000} TEGRA_SID_AFI>,” from “smmu” node
but all these string in the certain node is not available or already comment out in the extracted_from_dtb.dts
If someone know is where is location in the dts file for TX2 R28.1 to find the node and string using to disable SMMU ? very appreciate your can sharing your operate step on TX2 (R.28.1)
Beware that although the “/boot” dtb is probably the same as what was installed, this is a leftover behavior and this particular dtb no longer has any effect (or at least not the effect you will probably think of). During flash on R28.1 a TX2 gets this copied into “/dev/mmcblk0p15”. The flash process (see the previous posts on replacing the dtb in the flash subdirectory) will put this in place in the right spot.
Assuming your dtb is in place, this flashes only the kernel-dtb partition (mmcblk0p15 in R28.1):
sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1
After boot you should see if your changes are reflected in the content of “/proc/device-tree/”…this is the final sum total effect of all boot changes when the kernel takes over. You can browse directly, or turn into a dts for diff:
dtc -I fs -O dts -o extracted.dts /proc/device-tree
Hello Linuxdev
Thanks too much for your replay.
I am using TX2 board with R.28.1 package.
extracted system dts file using command: dtc -I fs -O dts -o extracted.dts /proc/device-tree
but in the extracted.dts file I can not find the string which vydias post as:
To disable SMMU . we need to do:
remove：
- “#stream-id-cells = <1>;” from “tegra_pcie” node
- “<&{/pcie-controller@10003000} TEGRA_SID_AFI>,” from “smmu” node
we can not find “tegra_pcie” node , can not find “TEGRA_SID_AFI” string
we need to modify the dts here by removing above string and recompile it to new dtb file and then
sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1 // flash new dtb into board
We just can not find the remove string needed to disable SMMU on TX2 with R.28.1
if someone knows which line in dts file to remove is need to disable SMMU for TX2 R.28.1
Please sharing in this post here. I think many developer on TX2 r.28.1 met this DMA issue recently.
Thanks in advance.
I see a topic on this:
[url]Jetson TX2, ioctl(dev->contfd, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU) hang up. - Jetson TX2 - NVIDIA Developer Forums
…however, I think this was probably from an earlier release…it might or might not apply to the current kernel.
In a device tree extracted from a default R28.1 TX2 install shows the “#stream-id-cells = <1>;” within the “pcie-controller@10003000” node. I am thinking perhaps the other line to remove is simply already removed and perhaps not now relevant. You may be seeing information which is outdated.
Someone else will need to verify the exact edits needed under R28.1/TX2 for this.
Hi all
I made it work to disable SMMU on TX2 R.28.1
thanks for you help.
My method is to edit the extracted dts file
removed “#stream-id-cells = <0x1>;” from the pcie-controller@10003000 node.
Re-compile dts to dtb : sudo dtc -I dts -O dtb -o tegra186-quill-p3310-1000-c03-00-base.dtb extracted.dts
Copy new tegra186-quill-p3310-1000-c03-00-base.dtb into PC linux : JetPack3.2/64_TX2/Linux_for_Tegra/kernel/dtb/ folder to replace old one.
sudo ./flash.sh -k kernel-dtb jetson-tx2 mmcblk0p1
reboot TX2 board. Check “/sys/kernel/debug/12000000.iommu/masters” and confirm there is no pcie related entries. Smmu is disabled after flash new dtb.
Hi,
when I edited and compiled dts extracted from system, where “<&{/pcie-controller@10003000} TEGRA_SID_AFI>,” line in smmu node is missing, I got error connected with 12000000.iommu in dmesg log. System appeared to work, but I decided to compile kernel and dtb from nvidia’s sources. In those sources you can find the above mentioned line and after applying this dtb the error in dmesg log didn’t appear. Further, if you extract this dtb from system, the line is missing again. I do not know much about device tree, but I found it as a better way of using self-compiled dtb.
Btw, I don’t see a reason for disabling SMMU, everything should work with SMMU enabled.
Hi Dusancerhaty
Disable SMMU is to make linux virt_to_phy function call return correct physical address and used for DMA or other mmap operation. If you are not using this kind of operation. leave SMMU enabled is suggested because Nvidia may use smmu to implement other task for some good reason.