Jetson Orin Nano 使用内核编译禁用IOMMU失败。

第一步:我修改了目录Linux_for_Tegra/source/public/kernel/kernel-5.10/arch/arm64/config 下的defconfig文件,修改如下
image

第二步:我修改了目录Linux_for_Tegra/source/public/hardware/nvidia/soc/t23x/kernel-dts/tegra234-soctegra34-soc-pcie.dtsi文件,把节点 /* C5 X8 */ pcie_c5_ep: pcie_ep@141a0000 中的iommu相关内容注释了。

 /* C5 X8 */
        pcie_c5_ep: pcie_ep@141a0000 {
                compatible = "nvidia,tegra234-pcie-ep", "snps,dw-pcie";
                power-domains = <&bpmp TEGRA234_POWER_DOMAIN_PCIEX8A>;
                reg = <0x00 0x141a0000 0x0 0x00020000     /* appl registers (128K)      */
                        0x00 0x3a040000 0x0 0x00040000    /* iATU_DMA reg space (256K)  */
                        0x00 0x3a080000 0x0 0x00040000    /* DBI space (256K)           */
                        0x27 0x40000000 0x4 0x00000000>;  /* Address Space (16G)        */
                reg-names = "appl", "atu_dma", "dbi", "addr_space";

                #address-cells = <3>;
                #size-cells = <2>;

                status = "disabled";

                num-lanes = <8>;

                clocks = <&bpmp_clks TEGRA234_CLK_PEX1_C5_CORE>,
                         <&bpmp_clks TEGRA234_CLK_PEX1_C5_CORE_M>;
                clock-names = "core", "core_m";

                resets = <&bpmp_resets TEGRA234_RESET_PEX1_CORE_5_APB>,
                         <&bpmp_resets TEGRA234_RESET_PEX1_CORE_5>;
                reset-names = "apb", "core";

                pinctrl-names = "default";
                pinctrl-0 = <&pex_rst_c5_in_state>;

                interrupts = <0 53 0x04>;       /* controller interrupt */
                interrupt-names = "intr";

                nvidia,dvfs-tbl = < 204000000 204000000 204000000  204000000
                                    204000000 204000000 204000000  665600000
                                    204000000 204000000 665600000  1600000000
                                    204000000 665600000 1600000000 2133000000 >;

                nvidia,host1x = <&host1x>;
                nvidia,enable-ext-refclk;
                nvidia,max-speed = <4>;
                nvidia,bar0-size = <0x100000>;  /* 1 MB */
                nvidia,device-id = /bits/ 16 <0x229B>;
                nvidia,controller-id = <&bpmp 0x5>;
                nvidia,aux-clk-freq = <0x13>;
                nvidia,disable-aspm-states = <0xf>;
                nvidia,aspm-cmrt = <0x3C>;
                nvidia,aspm-pwr-on-t = <0x14>;
                nvidia,aspm-l0s-entrance-latency = <0x3>;
                nvidia,aspm-l1-entrance-latency = <0x5>;

                num-ib-windows = <2>;
                num-ob-windows = <8>;

                //iommus = <&smmu_niso0 TEGRA_SID_NISO0_PCIE5>;
                //iommu-map = <0x0 &smmu_niso0 TEGRA_SID_NISO0_PCIE5 0x1000>;
                msi-parent = <&gic_v2m TEGRA_SID_NISO0_PCIE5>;
                msi-map = <0x0 &gic_v2m TEGRA_SID_NISO0_PCIE5 0x1000>;
                //dma-coherent;
                //iommu-map-mask = <0x0>;

                nvidia,cfg-link-cap-l1sub = <0x1c4>;
                nvidia,cap-pl16g-status = <0x174>;
                nvidia,cap-pl16g-cap-off = <0x188>;
                nvidia,event-cntr-ctrl = <0x1d8>;
                nvidia,event-cntr-data = <0x1dc>;
                nvidia,dl-feature-cap = <0x30c>;
                nvidia,ptm-cap-off = <0x318>;

                interconnects = <&tegra_icc TEGRA_ICC_PCIE_5 &tegra_icc TEGRA_ICC_PRIMARY>;
                interconnect-names = "icc_bwmgr";

                nvidia,bpmp = <&bpmp 5>;
                nvidia,aspm-cmrt-us = <60>;
                nvidia,aspm-pwr-on-t-us = <20>;
                nvidia,aspm-l0s-entrance-latency-us = <3>;
        };

第三步:进行内核编译,步骤如下:
3.1 配置编译环境变量
export CROSS_COMPILE_AARCH64_PATH=/usr
export CROSS_COMPILE_AARCH64=/usr/bin/aarch64-linux-gnu-
3.2 创建内核输出文件夹
mkdir kernel_out
3.3 编译内核
Linux_for_Tegra/source/public$ ./nvbuild.sh -o $PWD/kernel_out
3.4 将新编译好的 Image 和 dtb 文件替换至烧录位置
cp Image ~/Linux_for_Tegra/kernel/Image
cp dts/* ~/Linux_for_Tegra/kernel/dtb/ -
3.5 安装 kernel modules
sudo make ARCH=arm64 O=../../kernel_out modules_install INSTALL_MOD_PATH=~/kernel_moudules/
3.6 打包modules
~/kernel_moudules/$ sudo tar --owner root --group root -cjf kernel_supplements.tbz2 lib/modules
3.7 替换kernel_supplements.tbz2
sudo cp kernel_supplements.tbz2 ~/Linux_for_Tegra/kernel_supplements.tbz2
3.8 执行 apply_binaries.sh 脚本
~Linux_for_Tegra$ sudo ./apply_binaries.sh
3.9 Jetson Orin Nano 触发recovery模式,烧录固件
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal

但是最后我在目标机器/proc/device-tree/pcie@140a0000/下仍然能够看到iommu,如下:

请问是我哪一步操作出现了问题吗?

先不管中間過程的部分. 你可以先用dtc把你build好的dtb轉回dts檢查到底刪掉的內容還在不在

好的。 等我尝试下再回复您。

我对目录Linux_for_Tegra/source/public/kernel_out/arch/arm64/boot/dts/nvidia下的dtb文件进行反编译之后,查看其中一个tegra234-p3701-0000-as-p3701-0004-p3737-0000.dts文件发现 并没有Iommu的选项,也就是说我已经关闭了吗?

pcie_ep@141a0000 {
		compatible = "nvidia,tegra234-pcie-ep\0snps,dw-pcie";
		power-domains = <0x04 0x05>;
		reg = <0x00 0x141a0000 0x00 0x20000 0x00 0x3a040000 0x00 0x40000 0x00 0x3a080000 0x00 0x40000 0x27 0x40000000 0x04 0x00>;
		reg-names = "appl\0atu_dma\0dbi\0addr_space";
		#address-cells = <0x03>;
		#size-cells = <0x02>;
		status = "disabled";
		num-lanes = <0x08>;
		clocks = <0x04 0xe1 0x04 0xea>;
		clock-names = "core\0core_m";
		resets = <0x04 0x82 0x04 0x81>;
		reset-names = "apb\0core";
		pinctrl-names = "default";
		pinctrl-0 = <0x0f>;
		interrupts = <0x00 0x35 0x04>;
		interrupt-names = "intr";
		nvidia,dvfs-tbl = <0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0x27ac4000 0xc28cb00 0xc28cb00 0x27ac4000 0x5f5e1000 0xc28cb00 0x27ac4000 0x5f5e1000 0x7f22ff40>;
		nvidia,host1x = <0x10>;
		nvidia,enable-ext-refclk;
		nvidia,max-speed = <0x04>;
		nvidia,bar0-size = <0x100000>;
		nvidia,device-id = [22 9b];
		nvidia,controller-id = <0x04 0x05>;
		nvidia,aux-clk-freq = <0x13>;
		nvidia,disable-aspm-states = <0x0f>;
		nvidia,aspm-cmrt = <0x3c>;
		nvidia,aspm-pwr-on-t = <0x14>;
		nvidia,aspm-l0s-entrance-latency = <0x03>;
		nvidia,aspm-l1-entrance-latency = <0x05>;
		num-ib-windows = <0x02>;
		num-ob-windows = <0x08>;
		msi-parent = <0x11 0x14>;
		msi-map = <0x00 0x11 0x14 0x1000>;
		nvidia,cfg-link-cap-l1sub = <0x1c4>;
		nvidia,cap-pl16g-status = <0x174>;
		nvidia,cap-pl16g-cap-off = <0x188>;
		nvidia,event-cntr-ctrl = <0x1d8>;
		nvidia,event-cntr-data = <0x1dc>;
		nvidia,dl-feature-cap = <0x30c>;
		nvidia,ptm-cap-off = <0x318>;
		interconnects = <0x12 0x0f 0x12 0x00>;
		interconnect-names = "icc_bwmgr";
		nvidia,bpmp = <0x04 0x05>;
		nvidia,aspm-cmrt-us = <0x3c>;
		nvidia,aspm-pwr-on-t-us = <0x14>;
		nvidia,aspm-l0s-entrance-latency-us = <0x03>;
		vddio-pex-ctl-supply = <0x13>;
		reset-gpios = <0x14 0xb9 0x01>;
		nvidia,refclk-select-gpios = <0x14 0x7c 0x00>;
		phys = <0x15 0x16 0x17 0x18 0x19 0x1a 0x1b 0x1c>;
		phy-names = "p2u-0\0p2u-1\0p2u-2\0p2u-3\0p2u-4\0p2u-5\0p2u-6\0p2u-7";
		vpcie3v3-supply = <0x1d>;
		vpcie12v-supply = <0x1e>;
		phandle = <0x2d1>;
	};

pcie_ep@141e0000 {
		compatible = "nvidia,tegra234-pcie-ep\0snps,dw-pcie";
		power-domains = <0x04 0x10>;
		reg = <0x00 0x141e0000 0x00 0x20000 0x00 0x3e040000 0x00 0x40000 0x00 0x3e080000 0x00 0x40000 0x2e 0x40000000 0x04 0x00>;
		reg-names = "appl\0atu_dma\0dbi\0addr_space";
		#address-cells = <0x03>;
		#size-cells = <0x02>;
		status = "disabled";
		num-lanes = <0x08>;
		clocks = <0x04 0xab 0x04 0xf4>;
		clock-names = "core\0core_m";
		resets = <0x04 0x0f 0x04 0x0e>;
		reset-names = "apb\0core";
		pinctrl-names = "default";
		pinctrl-0 = <0x20>;
		interrupts = <0x00 0x162 0x04>;
		interrupt-names = "intr";
		nvidia,dvfs-tbl = <0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0x27ac4000 0xc28cb00 0xc28cb00 0x27ac4000 0x5f5e1000 0xc28cb00 0x27ac4000 0x5f5e1000 0x7f22ff40>;
		nvidia,enable-ext-refclk;
		nvidia,max-speed = <0x04>;
		nvidia,bar0-size = <0x100000>;
		nvidia,device-id = [22 9b];
		nvidia,controller-id = <0x04 0x07>;
		nvidia,aux-clk-freq = <0x13>;
		nvidia,disable-aspm-states = <0x0f>;
		nvidia,aspm-cmrt = <0x3c>;
		nvidia,aspm-pwr-on-t = <0x14>;
		nvidia,aspm-l0s-entrance-latency = <0x03>;
		nvidia,aspm-l1-entrance-latency = <0x05>;
		num-ib-windows = <0x02>;
		num-ob-windows = <0x08>;
		iommus = <0x21 0x08>;
		iommu-map = <0x00 0x21 0x08 0x1000>;
		msi-parent = <0x11 0x08>;
		msi-map = <0x00 0x11 0x08 0x1000>;
		dma-coherent;
		iommu-map-mask = <0x00>;
		nvidia,cfg-link-cap-l1sub = <0x1c4>;
		nvidia,cap-pl16g-status = <0x174>;
		nvidia,cap-pl16g-cap-off = <0x188>;
		nvidia,event-cntr-ctrl = <0x1d8>;
		nvidia,event-cntr-data = <0x1dc>;
		nvidia,dl-feature-cap = <0x30c>;
		nvidia,ptm-cap-off = <0x318>;
		interconnects = <0x12 0x11 0x12 0x00>;
		interconnect-names = "icc_bwmgr";
		nvidia,bpmp = <0x04 0x07>;
		nvidia,aspm-cmrt-us = <0x3c>;
		nvidia,aspm-pwr-on-t-us = <0x14>;
		nvidia,aspm-l0s-entrance-latency-us = <0x03>;
		vddio-pex-ctl-supply = <0x13>;
		phandle = <0x2d3>;
	};

请问,我在设备中怎样确认iommu已经关闭了?

你得先搞清楚自己到底在用哪一個device tree. 不是這樣盲目的隨便找一個轉
如果你不知道要找哪一個, 你先正常開機一次之後用dmesg |grep dts.

通过查看是文件tegra234-p3767-0003-p3768-0000-a0.dts


文件tegra234-p3767-0003-p3768-0000-a0.dts中PCIE节点信息如下:

pcie_ep@141a0000 {
		compatible = "nvidia,tegra234-pcie-ep\0snps,dw-pcie";
		power-domains = <0x02 0x05>;
		reg = <0x00 0x141a0000 0x00 0x20000 0x00 0x3a040000 0x00 0x40000 0x00 0x3a080000 0x00 0x40000 0x27 0x40000000 0x04 0x00>;
		reg-names = "appl\0atu_dma\0dbi\0addr_space";
		#address-cells = <0x03>;
		#size-cells = <0x02>;
		status = "disabled";
		num-lanes = <0x08>;
		clocks = <0x02 0xe1 0x02 0xea>;
		clock-names = "core\0core_m";
		resets = <0x02 0x82 0x02 0x81>;
		reset-names = "apb\0core";
		pinctrl-names = "default";
		pinctrl-0 = <0x47>;
		interrupts = <0x00 0x35 0x04>;
		interrupt-names = "intr";
		nvidia,dvfs-tbl = <0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0x27ac4000 0xc28cb00 0xc28cb00 0x27ac4000 0x5f5e1000 0xc28cb00 0x27ac4000 0x5f5e1000 0x7f22ff40>;
		nvidia,host1x = <0x48>;
		nvidia,enable-ext-refclk;
		nvidia,max-speed = <0x04>;
		nvidia,bar0-size = <0x100000>;
		nvidia,device-id = [22 9b];
		nvidia,controller-id = <0x02 0x05>;
		nvidia,aux-clk-freq = <0x13>;
		nvidia,disable-aspm-states = <0x0f>;
		nvidia,aspm-cmrt = <0x3c>;
		nvidia,aspm-pwr-on-t = <0x14>;
		nvidia,aspm-l0s-entrance-latency = <0x03>;
		nvidia,aspm-l1-entrance-latency = <0x05>;
		num-ib-windows = <0x02>;
		num-ob-windows = <0x08>;
		msi-parent = <0x49 0x14>;
		msi-map = <0x00 0x49 0x14 0x1000>;
		nvidia,cfg-link-cap-l1sub = <0x1c4>;
		nvidia,cap-pl16g-status = <0x174>;
		nvidia,cap-pl16g-cap-off = <0x188>;
		nvidia,event-cntr-ctrl = <0x1d8>;
		nvidia,event-cntr-data = <0x1dc>;
		nvidia,dl-feature-cap = <0x30c>;
		nvidia,ptm-cap-off = <0x318>;
		interconnects = <0x4a 0x0f 0x4a 0x00>;
		interconnect-names = "icc_bwmgr";
		nvidia,bpmp = <0x02 0x05>;
		nvidia,aspm-cmrt-us = <0x3c>;
		nvidia,aspm-pwr-on-t-us = <0x14>;
		nvidia,aspm-l0s-entrance-latency-us = <0x03>;
		phandle = <0x2ed>;
	};

	pcie_ep@141e0000 {
		compatible = "nvidia,tegra234-pcie-ep\0snps,dw-pcie";
		power-domains = <0x02 0x10>;
		reg = <0x00 0x141e0000 0x00 0x20000 0x00 0x3e040000 0x00 0x40000 0x00 0x3e080000 0x00 0x40000 0x2e 0x40000000 0x04 0x00>;
		reg-names = "appl\0atu_dma\0dbi\0addr_space";
		#address-cells = <0x03>;
		#size-cells = <0x02>;
		status = "disabled";
		num-lanes = <0x08>;
		clocks = <0x02 0xab 0x02 0xf4>;
		clock-names = "core\0core_m";
		resets = <0x02 0x0f 0x02 0x0e>;
		reset-names = "apb\0core";
		pinctrl-names = "default";
		pinctrl-0 = <0x4c>;
		interrupts = <0x00 0x162 0x04>;
		interrupt-names = "intr";
		nvidia,dvfs-tbl = <0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0xc28cb00 0x27ac4000 0xc28cb00 0xc28cb00 0x27ac4000 0x5f5e1000 0xc28cb00 0x27ac4000 0x5f5e1000 0x7f22ff40>;
		nvidia,enable-ext-refclk;
		nvidia,max-speed = <0x04>;
		nvidia,bar0-size = <0x100000>;
		nvidia,device-id = [22 9b];
		nvidia,controller-id = <0x02 0x07>;
		nvidia,aux-clk-freq = <0x13>;
		nvidia,disable-aspm-states = <0x0f>;
		nvidia,aspm-cmrt = <0x3c>;
		nvidia,aspm-pwr-on-t = <0x14>;
		nvidia,aspm-l0s-entrance-latency = <0x03>;
		nvidia,aspm-l1-entrance-latency = <0x05>;
		num-ib-windows = <0x02>;
		num-ob-windows = <0x08>;
		iommus = <0x4d 0x08>;
		iommu-map = <0x00 0x4d 0x08 0x1000>;
		msi-parent = <0x49 0x08>;
		msi-map = <0x00 0x49 0x08 0x1000>;
		dma-coherent;
		iommu-map-mask = <0x00>;
		nvidia,cfg-link-cap-l1sub = <0x1c4>;
		nvidia,cap-pl16g-status = <0x174>;
		nvidia,cap-pl16g-cap-off = <0x188>;
		nvidia,event-cntr-ctrl = <0x1d8>;
		nvidia,event-cntr-data = <0x1dc>;
		nvidia,dl-feature-cap = <0x30c>;
		nvidia,ptm-cap-off = <0x318>;
		interconnects = <0x4a 0x11 0x4a 0x00>;
		interconnect-names = "icc_bwmgr";
		nvidia,bpmp = <0x02 0x07>;
		nvidia,aspm-cmrt-us = <0x3c>;
		nvidia,aspm-pwr-on-t-us = <0x14>;
		nvidia,aspm-l0s-entrance-latency-us = <0x03>;
		vddio-pex-ctl-supply = <0x4e>;
		phandle = <0x2ef>;
	};

这样能说明关闭了吗?

我覺得這裡真正的問題應該是你到底想要關什麼…

比方說pcie_ep@141a0000 這代表的是PCIe C5的 EP controller… 但C5在Orin NX/Nano上根本就沒有在使用… 我不太確定你一直貼這個然後問我說關掉了是要做什麼…

关闭IOMMU的原因是:我的项目要外接一个PCIE的采集卡,这个采集卡的驱动是不支持IOMMU的。所以我想禁用Jetson 上的IOMMU。这

不是… 我知道你要關IOMMU, 我的意思是你到底要關哪一個PCIe的IOMMU.

Jetson上面有很多個pcie controller…

感谢您的指导;很抱歉,我没有表达清楚我的意思 。
我的PCIE采集卡插在了M2 KEY M的插槽上,如下图:

经过查阅NVIDIA官网资料
下图中C0,C4,C6,C10是控制PCIE LAN WIDTH的节点 ,所以我如果要关闭IOMMU的话,是否是关闭C0,C4,C6,C10下的IOMMU节点?

不知道 我的理解是否正确?

如果是Orin Nano devkit的M.2 key M, 那麼是C4.

C4對應的node是pcie@14160000. 這個才是你應該要改的

好的。 我明白了。
在文件tegra234-soc-pcie.dtsi中C4的状态是disabled.表明了,这个节点的状态是默认关闭的,那么我是不是就没有必要注释掉iommu的属性了?

另外 C0,C4,C6,C10 都是控制PCIE LAN WIDTH 的节点。请问这样节点有什么区别呢?

那么我是不是就没有必要注释掉iommu的属性了?

不是…這個node disabled/enabled是代表這整個controller有沒有打開. 跟iommu有沒有開關是兩回事…
這node不打開你的device一定不會link up…

另外 C0,C4,C6,C10 都是控制PCIE LAN WIDTH 的节点。请问这样节点有什么区别呢?

這些controller跟硬體有對應關係 不是每個controller在Orin Nano上都存在. 如果你對這些不清楚的話可以參考design guide文件

好的。谢谢您的回复和支持,帮我解决了问题。祝您生活愉快。