I am running PCIe on Jetson AGX Xavier (root complex), with a Xilinx FPGA based end point device.
We are currently using end point bus mastering to write data directly from the end point device to the Xavier’s memory.
We are successfully able to do this when SMMU is enabled. However, when I disable SMMU, I run into an issue: I receive MSI on Xavier (meaning that PCIe bus has written the data to Xavier’s memory), but when I check the data in the Xavier’s memory, the data is not present. The memory contains all zeros instead of the PCIe payload (it was initialized to all zeros when the memory was allocated).
What could be causing this behaviour? Everything works when SMMU is enabled, but with SMMU disabled, my RX memory buffer doesn’t receive any PCIe data.
I am running Jetpack 5.1. (I need the SMMU disabled for some other reason, so re-enabling the SMMU is not really an option). For reference, I disabled SMMU using this method:
Yes, this is still an issue. I am not receiving any data at all in AGX Xavier’s memory, which is acting as root port. I am using a Xilinx FPGA based end point, which DMA’s data directly to root port’s memory. I can see PCIe data coming out of the FPGA via integrated logic analyzer, but my buffers on AGX Xavier side are empty. (I receive MSI, so I was expecting the buffers to contain the data).
Can you please specify what information is required to debug this? I have a kernel module running on AGX Xavier which allocates the memory using dma_map_single(). I am happy to share the relevant portion of the kernel module code privately if it would help.
Note: As mentioned in my first comment, there is no problem when I enable SMMU, so I think the kernel module code is ok. Also note that at this moment I can’t re-enable SMMU, because I run into this problem if I do:
I am happy to re-enable the SMMU if I can find a solution to the issue mentioned in the linked post above.
Please let me know any further info that is required.