Jetson AGX Orig disable PCIe IOMMU error

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi, I want to extend Jetson AGX Orig with Xilinx FX600 on PCIe x8/x16. I need to disable PCIe IOMMU to compatible with original code.

I disble all PCIe controller nodes that modification as following:

diff --git a/hardware/nvidia/soc/t23x/kernel-dts/tegra234-soc/tegra234-soc-pcie.dtsi b/../../../hardware/nvidia/soc/t23x/kernel-dts/tegra234-soc/tegra234-soc-pcie.dtsi
old mode 100644
new mode 100755
index 40d3284..f0974e4
--- a/hardware/nvidia/soc/t23x/kernel-dts/tegra234-soc/tegra234-soc-pcie.dtsi
+++ b/../../../hardware/nvidia/soc/t23x/kernel-dts/tegra234-soc/tegra234-soc-pcie.dtsi
@@ -465,12 +465,12 @@
                num-ib-windows = <2>;
                num-ob-windows = <8>;
 
-               //iommus = <&smmu_niso0 TEGRA_SID_NISO0_PCIE5>;
-               //iommu-map = <0x0 &smmu_niso0 TEGRA_SID_NISO0_PCIE5 0x1000>;
+               iommus = <&smmu_niso0 TEGRA_SID_NISO0_PCIE5>;
+               iommu-map = <0x0 &smmu_niso0 TEGRA_SID_NISO0_PCIE5 0x1000>;
                msi-parent = <&gic_v2m TEGRA_SID_NISO0_PCIE5>;
                msi-map = <0x0 &gic_v2m TEGRA_SID_NISO0_PCIE5 0x1000>;
-               //dma-coherent;
-               //iommu-map-mask = <0x0>;
+               dma-coherent;
+               iommu-map-mask = <0x0>;
 
                nvidia,cfg-link-cap-l1sub = <0x1c4>;
                nvidia,cap-pl16g-status = <0x174>;

But i get those error message:

x@tegra-ubuntu:~$ [   28.640756] arm-smmu 8000000.iommu: Unexpected global fault, this could be serious
[   28.648555] arm-smmu 8000000.iommu:  GFSR 0x80000002, GFSYNR0 0x00000000, GFSYNR1 0x00000c05, GFSYNR2 0x0000000                                                                                                                           0
[   28.659023] arm-smmu 8000000.iommu: Unexpected global fault, this could be serious
[   28.666803] arm-smmu 8000000.iommu:  GFSR 0x80000002, GFSYNR0 0x00000000, GFSYNR1 0x00000405, GFSYNR2 0x0000000                                                                                                                           0
[   28.678425] mc-err: (255) csr_pcie1r: EMEM address decode error
[   28.684545] mc-err:   status = 0x200640da; hi_addr_reg = 0x000000ff addr = 0xffffffff00
[   28.692812] mc-err:   secure: yes, access-type: read
[   28.697988] pcieport 0001:00:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Recei                                                                                                                           ver ID)
[   28.709037] pcieport 0001:00:00.0:   device [10de:229e] error status/mask=00040000/00400000
[   28.717647] pcieport 0001:00:00.0:    [18] MalfTLP                (First)
[   28.724702] pcieport 0001:00:00.0: AER:   TLP Header: 00000003 01000178 fffffffc 00000000

Is this way too rude cause error? which PCIe controller node need to disable?

3

which PCIe controller node need to disable?

Which PCIe controller are you using for your Xilinx FX600…

4

Hello ， I diable C5 X8 and C7 X8 IOMMU. But I get such error messages ,when transport some data.

[  120.158998] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740001000, fsynr=0x6b0001, cbfrsynra=0x814, cb=6
[  120.171407] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740000000, fsynr=0x310001, cbfrsynra=0x814, cb=6
[  120.183865] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740001000, fsynr=0x6b0001, cbfrsynra=0x814, cb=6
[  120.198936] mc-err: (255) csr_pcie5r1: EMEM address decode error
[  120.205117] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740001000, fsynr=0x6b0001, cbfrsynra=0x814, cb=6
[  120.205182] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740000000, fsynr=0x310001, cbfrsynra=0x814, cb=6
[  120.230033] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740001000, fsynr=0x6b0001, cbfrsynra=0x814, cb=6
[  120.230097] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740000000, fsynr=0x310001, cbfrsynra=0x814, cb=6
[  120.242513] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740001000, fsynr=0x6b0001, cbfrsynra=0x814, cb=6
[  120.242577] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740000000, fsynr=0x310001, cbfrsynra=0x814, cb=6
[  120.269879] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x740001000, fsynr=0x6b0001, cbfrsynra=0x814, cb=6
[  120.297367] mc-err:   status = 0x200640ef; hi_addr_reg = 0x000000ff addr = 0xffffffff00
[  120.308290] mc-err:   secure: yes, access-type: read
[  120.321858] mc-err: (255) csr_pcie5r1: EMEM address decode error
[  120.328053] mc-err:   status = 0x200640ef; hi_addr_reg = 0x000000ff addr = 0xffffffff00
[  120.336297] mc-err:   secure: yes, access-type: read
[  120.361862] mc-err: (255) csr_pcie5r1: EMEM address decode error
[  120.368055] mc-err:   status = 0x200640ef; hi_addr_reg = 0x000000ff addr = 0xffffffff00
[  120.376298] mc-err:   secure: yes, access-type: read
[  120.399305] mc-err: (255) csr_pcie5r1: EMEM address decode error
[  120.405499] mc-err:   status = 0x200640ef; hi_addr_reg = 0x000000ff addr = 0xffffffff00
[  120.413742] mc-err:   secure: yes, access-type: read
5

What is the result of /sys/kernel/debug/12000000.iommu/masters after you apply the patch in device tree?

6

image
image964×150 3.78 KB

7

Have some problem?

8

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Sorry for the late response, is this still an issue to support? Thanks