PCIe problem after jetpack update

Robotics & Edge Computing Jetson Systems Jetson TX1
1

Hi,

I have FPGA based acquisition device connected to TX1 development board through PCIe. Driver on TX1 side
is very simple piece of code. Everything worked fine on older Jetpack

# R24 (release), REVISION: 1.0, GCID: 7164062, BOARD: t210ref, EABI: aarch64, DATE: Tue May 17 23:37:30 UTC 2016

but after upgrade to the newest 3.0 driver started to operate very odd, data synchronisation done
after transfer works only partially, some of bytes are correctly synchronised, some not, and
every time I execute function

dma_sync_single_for_cpu

or

dma_sync_single_for_device

I receive warning in

dmesg

[  585.453299] WARNING: at /dvs/git/dirty/git-master_linux/kernel/arch/arm64/mm/dma-mapping.c:2261 arm_iommu_sync_single_for_device+0x84/0xac()
[  585.459586] WARNING: at /dvs/git/dirty/git-master_linux/kernel/arch/arm64/mm/dma-mapping.c:2244 arm_iommu_sync_single_for_cpu+0x84/0xac()

Full warning code

[   91.779499] ------------[ cut here ]------------
[   91.779510] WARNING: at /dvs/git/dirty/git-master_linux/kernel/arch/arm64/mm/dma-mapping.c:2244 arm_iommu_sync_single_for_cpu+0x84/0xac()
[   91.779514] Modules linked in: zonasens(O) bnep bcmdhd cfg80211 bluedroid_pm
[   91.779533] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W  O 3.10.96-tegra #1
[   91.779537] Call trace:
[   91.779542] [<ffffffc000089df4>] dump_backtrace+0x0/0xf4
[   91.779547] [<ffffffc00008a0f4>] show_stack+0x14/0x1c
[   91.779552] [<ffffffc00032a59c>] dump_stack+0x20/0x28
[   91.779558] [<ffffffc0000a72e4>] warn_slowpath_common+0x78/0x9c
[   91.779564] [<ffffffc0000a73d0>] warn_slowpath_null+0x18/0x20
[   91.779569] [<ffffffc000099808>] arm_iommu_sync_single_for_cpu+0x84/0xac
[   91.779575] [<ffffffbffcfc2238>] irq_dma_done+0x64/0x90 [my_driver]
[   91.779582] [<ffffffc00012377c>] handle_irq_event_percpu+0xd8/0x22c
[   91.779587] [<ffffffc000123918>] handle_irq_event+0x48/0x74
[   91.779593] [<ffffffc000126314>] handle_simple_irq+0xac/0xc0
[   91.779598] [<ffffffc000122c34>] generic_handle_irq+0x28/0x40
[   91.779604] [<ffffffc000385608>] tegra_pcie_msi_irq+0xe8/0x168
[   91.779609] [<ffffffc00012377c>] handle_irq_event_percpu+0xd8/0x22c
[   91.779614] [<ffffffc000123918>] handle_irq_event+0x48/0x74
[   91.779619] [<ffffffc000126aec>] handle_fasteoi_irq+0xc4/0x10c
[   91.779624] [<ffffffc000122c34>] generic_handle_irq+0x28/0x40
[   91.779630] [<ffffffc0000859dc>] handle_IRQ+0x94/0xc8
[   91.779635] [<ffffffc0000813d4>] gic_handle_irq+0x74/0x194

The only difference I found is that extlinux loads some init ram in the newer version, but on the older not. Removing line which loads init ram ends up with system without usb, eth and wifi (but hdmi still works).

I understand staying with older jetpack is the only reasonable version for now ?

2

There have been some PCIe devices which don’t work with the IOMMU enabled, it sounds like that’s what you’ve run into. I’m not positive about which patch actually worked to disable the IOMMU though. Some of the info is found via this, though it isn’t exact:
[url]Google

Anyone know the exact patch for IOMMU disable on PCIe for the TX1?

3

Please apply following changes to disable SMMU for PCIe

--- a/arch/arm64/boot/dts/tegra210-soc-base.dtsi
+++ b/arch/arm64/boot/dts/tegra210-soc-base.dtsi
@@ -1252,6 +1252,7 @@
0x82000000 0 0x13000000 0x0 0x13000000 0 0x0d000000 /* non-prefetchable memory (208 MiB) */
0xc2000000 0 0x20000000 0x0 0x20000000 0 0x20000000>; /* prefetchable memory (512 MiB) */

+ iommus = <&smmu TEGRA_SWGROUP_AFI>;
status = "disabled";

pci@1,0 {

and

--- a/drivers/iommu/of_tegra-smmu.c
+++ b/drivers/iommu/of_tegra-smmu.c
@@ -166,7 +166,6 @@ u64 tegra_smmu_of_get_swgids(struct device *dev,
u64 fixup, swgids = 0;

if (dev_is_pci(dev)) {
- return SWGIDS_ERROR_CODE;
swgids = TEGRA_SWGROUP_BIT(AFI);
goto try_fixup;
}