"dma_alloc_coherent" return memory address can't to use for "virt_to_page" on Jetpack 3.2 TX1 kernel

Hi Guys,

I am porting our PCIe device driver to Jetpack 3.2 TX1.

When I use the API of “dma_alloc_coherent()” to allocate the memory, and use “virt_to_page()” to mapping the memory page.

It will get error page address.

Because, When I use this page address for “SetPageReserved()”, it will show the kernel exception.

But this code use on Jetpack 2.3.1, it will work.

Anyone have ideas ?

Thanks!!!
BR.

Alleen Wang

Older JetPack versions didn’t have SMMU enabled for PCIe whereas the newer ones have.
So, one quick way is to disable SMMU for PCIe, but better way is to fix the code in such a way that it would work for both scenarios

Hi Vidyas,

Thanks for your reply.

How to disable the SMMU on kernel ?

Disable CONFIG_TEGRA_IOMMU_SMMU ?

Thanks!!!
BR.

Alleen Wang

Please use the below patch to disable SMMU for PCIe

diff --git a/kernel-dts/tegra210-soc/tegra210-soc-base.dtsi b/kernel-dts/tegra210-soc/tegra210-soc-base.dtsi
index dacc13a19cb9..524aff2dbda3 100644
--- a/kernel-dts/tegra210-soc/tegra210-soc-base.dtsi
+++ b/kernel-dts/tegra210-soc/tegra210-soc-base.dtsi
@@ -1629,8 +1629,6 @@
         pinctrl-4 = <&pex_io_dpd_disable_state>;
         pinctrl-5 = <&pex_io_dpd_enable_state>;

-        iommus = <&smmu TEGRA_SWGROUP_AFI>;
-
         bus-range = <0x00 0xff>;
         #address-cells = <3>;
         #size-cells = <2>;

Hi Vidyas,

After I used your patch, it still has kernel exception.
When I call the “SetPageReserved()”, Kernel will show the exception …

[ 66.395260] Plx8000_NT: <========================================================>
[ 66.404197] Plx8000_NT: PLX 8000_NT driver v7.25 (64-bit)
[ 66.410099] Plx8000_NT: Supports Linux kernel v4.4.38
[ 66.415178] Plx8000_NT: Allocated global driver object (ffffffc07ba10e00)
[ 66.422043] Plx8000_NT: Registered driver (MajorID = 232)
[ 66.428107] Plx8000_NT: --------------------
[ 66.432889] Plx8000_NT: Probe: 87A0 10B5 [D0 01:00.0]
[ 66.438489] Plx8000_NT: Enabled PCI device
[ 66.442610] Plx8000_NT: Created Device (Plx8000_NT-0)
[ 66.447842] Plx8000_NT: Start: 87A0 10B5 [D0 01:00.0]
[ 66.452934] Plx8000_NT: Resource 00
[ 66.456713] Plx8000_NT: Type : Memory
[ 66.461202] Plx8000_NT: PCI BAR 0: 13000000
[ 66.466158] Plx8000_NT: Phys Addr: 13000000
[ 66.470796] Plx8000_NT: Size : 40000h (256KB)
[ 66.475944] Plx8000_NT: Property : Non-Prefetchable 32-bit
[ 66.482046] Plx8000_NT: Kernel VA: ffffff800da80000
[ 66.487359] Plx8000_NT: Resource 01
[ 66.491108] Plx8000_NT: Type : Memory
[ 66.495551] Plx8000_NT: PCI BAR 2: 2000000C
[ 66.500161] Plx8000_NT: Phys Addr: 20000000
[ 66.504772] Plx8000_NT: Size : 4000000h (64MB)
[ 66.509989] Plx8000_NT: Property : Prefetchable 64-bit
[ 66.516425] Plx8000_NT: Kernel VA: ffffff800db00000
[ 66.521782] Plx8000_NT: Device 87A0_10B5 = 8717 rev CA
[ 66.527174] Plx8000_NT: Device in NT STANDARD P2P mode
[ 66.532328] Plx8000_NT: NT port is LINK-side (NT base=3F000h)
[ 66.538116] Plx8000_NT: MSI enabled, change IRQ 112 --> 408
[ 66.543684] Plx8000_NT: Resource 02
[ 66.547431] Plx8000_NT: Type : Interrupt (MSI)
[ 66.552660] Plx8000_NT: IRQ : 408 [198h]
[ 66.557455] Plx8000_NT: MSI Addr : 0000_FD397000
[ 66.566000] Plx8000_NT: MSI Data : 0000
[ 66.570286] Plx8000_NT: Installed ISR for interrupt
[ 66.575220] Plx8000_NT: Attempt to allocate physical memory (8KB)
[ 66.639677] Unable to handle kernel paging request at virtual address 3ffffbcc2010040
[ 66.647519] pgd = ffffffc05c606000
[ 66.650921] [3ffffbcc2010040] *pgd=0000000000000000, *pud=0000000000000000
[ 66.657818] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 66.663375] Modules linked in: Plx8000_NT_dbg(O+) bcmdhd bluedroid_pm
[ 66.669827] CPU: 1 PID: 1905 Comm: insmod Tainted: G O 4.4.38 #1
[ 66.676943] Hardware name: jetson_tx1 (DT)
[ 66.681026] task: ffffffc0e81ad780 ti: ffffffc05c590000 task.ti: ffffffc05c590000
[ 66.688495] PC is at set_bit+0x18/0x30
[ 66.692250] LR is at Plx_dma_buffer_alloc+0x148/0x2c4 [Plx8000_NT_dbg]
[ 66.698760] pc : [] lr : [] pstate: 80000145
[ 66.706135] sp : ffffffc05c593b50
[ 66.709438] x29: ffffffc05c593b50 x28: ffffffbffcf43290
[ 66.714746] x27: ffffffbffcf3dd08 x26: ffffffbffcf41c4a
[ 66.720054] x25: 0000000000000030 x24: ffffffbffcf3d3d0
[ 66.725362] x23: ffffffbffcf3a320 x22: ffffffc0012dce30
[ 66.730669] x21: 0000004000000000 x20: ffffffc0e52b6f00
[ 66.735977] x19: ffffff8000401000 x18: 0000000000000010
[ 66.741283] x17: 0000000000000000 x16: 0000000000000000
[ 66.746590] x15: 0000000000000000 x14: 6d6d6f632d676e69
[ 66.751898] x13: 7070616d2d616d64 x12: 2f636972656e6567
[ 66.757204] x11: ffffffc001070f00 x10: 0000000000000890
[ 66.762511] x9 : ffffffc05c593920 x8 : ffffffc0e81ae070
[ 66.767817] x7 : 0000000000000020 x6 : 0000000000000000
[ 66.773124] x5 : 0000000000000400 x4 : ffffffc0e81ae078
[ 66.778430] x3 : 0000000000000400 x2 : 0000000000000001
[ 66.783738] x1 : 03ffffbcc2010040 x0 : 0000000000000000
[ 66.789043]
[ 66.790526] Process insmod (pid: 1905, stack limit = 0xffffffc05c590020)
[ 66.797209] Call trace:
[ 66.799651] [] set_bit+0x18/0x30
[ 66.804444] [] PlxPciPhysicalMemoryAllocate+0xb4/0x1c8 [Plx8000_NT_dbg]
[ 66.812620] [] init_module+0x17c/0x290 [Plx8000_NT_dbg]
[ 66.819396] [] do_one_initcall+0x110/0x1c4
[ 66.825042] [] do_init_module+0x64/0x1c4
[ 66.830515] [] load_module+0xc38/0x10d8
[ 66.835899] [] SyS_finit_module+0x98/0xb4
[ 66.841458] [] el0_svc_naked+0x24/0x28
[ 66.846991] —[ end trace 5e416e0aae312542 ]—

Thanks!!!
BR.

Alleen Wang

You mean, you observe this error before and after applying patch in #4 ?
Can you please confirm whether you see any PCIe entries in “/sys/kernel/debug/70019000.iommu/masters/” location?

Hi Vidyas,

Base on my test result.
When I used your patch, and I changed the flag “GFP_ATOMIC” to “GFP_KERNEL | __GFP_NOWARN”.
It can work fine.

Success case :
dma_alloc_coherent(
…,
…,
…,
GFP_KERNEL | __GFP_NOWARN, <— This work fine.
)

But when I used the flag “GFP_ATOMIC”, it still show exception.

Fail case :
dma_alloc_coherent(
…,
…,
…,
GFP_ATOMIC, <— Kernel show exception information.
)

I can see PCIe entries : “/sys/kernel/debug/70019000.iommu/masters/” .

Thanks!!!
BR.

Alleen Wang