PCIe access to reserved memory causes iommu fault

[dbrazeau1]

I’m trying to read/write to reserved memory from a PCI device, but when the PCI device tries to access reserve memory, I get an iommu error.

Here is the reserve memory entry I added to my device tree.

	reserved-memory {
		#address-cells = <0x02>;
		#size-cells = <0x02>;
		ranges;


		ntb_dma_buffer {
			reg = <0x1 0x00000000 0x0 0x08000000>; // 128 MB			
			no-map;
			status = "okay";
		};
	};

This is the error I’m seeing on a Orin NX running Jetson Linux R35.4.1 when the PCI device is attempting to DMA from reserved memory.

[  363.909202] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x100000000, fsynr=0x6c0001, cbfrsynra=0x13, cb=5
[  363.924244] mc-err: (255) csr_pcie4r: EMEM address decode error
[  363.930353] mc-err:   status = 0x200640e0; hi_addr_reg = 0x000000ff addr = 0xffffffff00
[  363.938600] mc-err:   secure: yes, access-type: read

I’ve also attempted to disable iommu for pcie4r by removing the lines from pcie@14160000 in the device tree.

		iommus = <0x06 0x13>;
		iommu-map = <0x00 0x06 0x13 0x1000>;
		dma-coherent;
		iommu-map-mask = <0x00>;

My attempt to disable iommu for the PCIe controller results in a different iommu fault when trying to read reserved memory.

[  808.754503] arm-smmu 12000000.iommu: Unexpected global fault, this could be serious
[  808.762424] arm-smmu 12000000.iommu:         GFSR 0x80000002, GFSYNR0 0x00000000, GFSYNR1 0x00000013, GFSYNR2 0x00000000
[  808.775625] mc-err: (255) csr_pcie4r: EMEM address decode error
[  808.781743] mc-err:   status = 0x200640e0; hi_addr_reg = 0x000000ff addr = 0xffffffff00
[  808.789984] mc-err:   secure: yes, access-type: read

What do I need to do to get this to work, and prevent an iommu fault when a PCI device is trying to access reserved memory?

[kayccc]

Sorry for the late response.
Is this still an issue to support? Any result can be shared?

[dbrazeau1]

I have not had any luck getting it to work by reserving memory in the device tree.

I ended up writing a small driver that uses dma_alloc_coherent() and the PCI device’s device handle to allocate memory on the Orin NX board, which can now be read/written to by the PCI device.