pci_map_sg sometimes leads to kernel-oops

mrjazz2 · November 25, 2016, 11:52am

I connected a Xilinx Kintex-7 FPGA over PCIe to the Jetson TX1 and want to exchange data with DMA. For configuring the DMA-controller I’m using the method pci_map_sg (with the pages returned by get_user_pages). It often works, but sometimes the system crashes within the execution of pci_map_sg. In the kern.log-file I can see that a kernel oops is happening in the smmu_iommu_map_sg-method. Do you have any idea what might be the problem?
This is the part of the kernel-log related to this error (“ql2pcie” is the name of my kernel module and “ql2test” my user-space test-app):

Nov 24 15:41:01 tegra-ubuntu kernel: [   67.730183] ql2: Attempting to call pci_map_sg.
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.741544] Unable to handle kernel paging request at virtual address 100005400
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.748849] pgd = ffffffc0fc7db000
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.752245] [100005400] *pgd=0000000000000000
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.756612] Internal error: Oops: 96000005 [#1] PREEMPT SMP
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.762177] Enter nvdumper_crash_setup_regs
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.766417] nvdumper: all registers are saved.
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.766443] nvdumper: all registers are saved.
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.766450] nvdumper: all registers are saved.
Nov 24 15:41:01 tegra-ubuntu kernel: [   67.779732] nvdumper: all registers are saved.
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.784170] Modules linked in: pci_tegra ql2pcie(O) bnep bcmdhd cfg80211 bluedroid_pm
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.792105] CPU: 0 PID: 1854 Comm: ql2test Tainted: G           O 3.10.96 #6
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.799143] task: ffffffc0ee509380 ti: ffffffc0a8240000 task.ti: ffffffc0a8240000
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.806620] PC is at smmu_iommu_map_sg+0x2ec/0x408
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.811404] LR is at smmu_iommu_map_sg+0x2e4/0x408
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.816188] pc : [<ffffffc00084dfac>] lr : [<ffffffc00084dfa4>] pstate: 200003c5
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.823572] sp : ffffffc0a8243b80
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.826882] x29: ffffffc0a8243de0 x28: 0000000100005400
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.832221] x27: 00000000e0000000 x26: ffffffc0a8240000
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.837554] x25: ffffffc0a61bf000 x24: 0000000000126179
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.842885] x23: 00000000000003ef x22: 0000000000000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.848217] x21: 0000000083fe0000 x20: ffffffc0fe26e800
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.853549] x19: 000000000000020f x18: 0000000000000a03
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.858881] x17: 0000000000412070 x16: 000000000000002e
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.864211] x15: 0000000000000067 x14: 0000000000000073
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.869543] x13: 000000000000006d x12: 000000000000005f
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.874874] x11: 0000000000000069 x10: 0000000000000063
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.880206] x9 : ffffffc0a82439a0 x8 : 0000000000000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.885536] x7 : 0000000000000553 x6 : 0000000080000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.890868] x5 : ffffffc0fe1983e8 x4 : 0000000080000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.896200] x3 : 6db6db6db6db6db7 x2 : ffffffc0d9dfac00
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.901531] x1 : 0000000100005400 x0 : 0000000100005400
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.906863]
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.906863] PC: 0xffffffc00084df2c:
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.911819] df2c  34000320 f9403fe0 b9400814 b9404b40 11000400 b9004b40 f9401033 b4000193
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.920115] df4c  d374cf15 f9400266 2a1b03e5 d2820004 aa1503e3 f94037e2 2a1403e1 f9400660
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.928407] df6c  d63f00c0 f8410e60 b5fffee0 b9404b40 51000400 b9004b40 f9400340 36080040
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.936699] df8c  940a2158 91000718 f10006d6 54000281 aa1c03e0 97eb4c4a aa0003fc b4000200
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.944984] dfac  f9400018 927ef718 d2c00880 8b000318 9343ff18 d28db6e0 f2b6db60 f2db6da0
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.953277] dfcc  f2edb6c0 9b007f18 b9400b80 b9400f96 0b160016 113ffed6 d34c7ed6 110006f7
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.961571] dfec  b9409be0 6b17001f 54fff861 b9409fe0 8b20cb39 b98093f3 d37ef661 aa1903e0
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.969859] e00c  97e12e1d f9405fe0 f94063e1 eb01001f 54000148 d0005220 f9456006 aa1303e5
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.978159]
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.978159] LR: 0xffffffc00084df24:
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.983117] df24  f9403be1 b9400820 34000320 f9403fe0 b9400814 b9404b40 11000400 b9004b40
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.991411] df44  f9401033 b4000193 d374cf15 f9400266 2a1b03e5 d2820004 aa1503e3 f94037e2
Nov 24 15:41:02 tegra-ubuntu kernel: [   67.999704] df64  2a1403e1 f9400660 d63f00c0 f8410e60 b5fffee0 b9404b40 51000400 b9004b40
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.008003] df84  f9400340 36080040 940a2158 91000718 f10006d6 54000281 aa1c03e0 97eb4c4a
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.016300] dfa4  aa0003fc b4000200 f9400018 927ef718 d2c00880 8b000318 9343ff18 d28db6e0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.024593] dfc4  f2b6db60 f2db6da0 f2edb6c0 9b007f18 b9400b80 b9400f96 0b160016 113ffed6
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.032894] dfe4  d34c7ed6 110006f7 b9409be0 6b17001f 54fff861 b9409fe0 8b20cb39 b98093f3
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.041188] e004  d37ef661 aa1903e0 97e12e1d f9405fe0 f94063e1 eb01001f 54000148 d0005220
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.049474]
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.049474] SP: 0xffffffc0a8243b00:
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.054431] 3b00  fe26e800 ffffffc0 83fe0000 00000000 00000000 00000000 000003ef 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.062721] 3b20  00126179 00000000 a61bf000 ffffffc0 a8240000 ffffffc0 e0000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.071016] 3b40  00005400 00000001 a8243de0 ffffffc0 0084dfa4 ffffffc0 a8243b80 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.079317] 3b60  0084dfac ffffffc0 200003c5 00000000 0084de08 ffffffc0 0084e06c ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.087617] 3b80  00000000 00000000 fe26e800 ffffffc0 83fe0000 00000000 d9dfaa00 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.095913] 3ba0  83fe0000 00000000 d9dfaa00 ffffffc0 00000000 00000000 83fe0000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.104208] 3bc0  00000010 00000000 ca97c3f0 ffffffc0 008499d0 ffffffc0 1fe31670 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.112504] 3be0  ee5098a0 ffffffc0 83fe0000 00000000 01300900 ffffffc0 fe1983e8 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.120793]
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.120793] X2: 0xffffffc0d9dfab80:
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.125750] ab80  04055198 ffffffbc 00000000 00001000 ffffffff ffffffff 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.134044] aba0  040551d0 ffffffbc 00000000 00001000 ffffffff ffffffff 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.142338] abc0  04055208 ffffffbc 00000000 00001000 ffffffff ffffffff 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.150629] abe0  04055240 ffffffbc 00000000 00001000 ffffffff ffffffff 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.158916] ac00  00005401 00000001 00000000 00000000 fc55f180 ffffffc0 00b4ca98 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.167208] ac20  00000009 00000000 00000000 00000000 00000000 00000000 d9dfac38 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.175502] ac40  d9dfac38 ffffffc0 d9dfac48 ffffffc0 d9dfac48 ffffffc0 d5d1af00 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.183805] ac60  00000001 00000000 d9dfac68 ffffffc0 d9dfac68 ffffffc0 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.192103]
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.192103] X5: 0xffffffc0fe198368:
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.197060] 8368  00000000 00000000 00000000 00000000 ff828a80 ffffffc0 fe198018 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.205353] 8388  00000007 00010001 053986f8 ffffffbc e0000000 e0000000 e0000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.213648] 83a8  fe2a6018 ffffffc0 fe1983b0 ffffffc0 fe1983b0 ffffffc0 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.221948] 83c8  00000000 00000000 00000000 00000000 00000000 00000000 ff828780 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.230238] 83e8  fe198018 ffffffc0 00000008 0e010e00 05398928 ffffffbc e0000000 e0000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.238535] 8408  e0000000 00000000 fe2a8018 ffffffc0 caab38f8 ffffffc0 fe26d838 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.246830] 8428  00030003 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.255127] 8448  ff82c380 ffffffc0 fe198018 ffffffc0 00000009 90cb90cb 05399568 ffffffbc
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.263432]
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.263432] X9: 0xffffffc0a8243920:
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.268389] 3920  00156640 ffffffc0 000003c0 00000000 00028020 ffffffc0 00200000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.276692] 3940  00000000 00000000 00008020 00000000 00000001 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.284989] 3960  012f5000 ffffffc0 00008020 00000000 00000001 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.293287] 3980  00000000 00000000 00000041 00000000 00157538 ffffffc0 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.301585] 39a0  0008120c ffffffc0 ffc0ba0c ffffffc0 0000020f 00000000 fe26e800 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.309870] 39c0  a8243b80 ffffffc0 0084dfac ffffffc0 00084c30 ffffffc0 012f5008 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.318156] 39e0  00000000 00000000 00081394 ffffffc0 00000000 ffffffff 00000000 00000001
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.326453] 3a00  00000000 00000000 00000000 00000001 00000340 00000000 012f4200 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.334745]
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.334745] X20: 0xffffffc0fe26e780:
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.339789] e780  2e2f2e2e 73612f2e 00383030 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.348077] e7a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.356365] e7c0  fe264a80 ffffffc0 fe276740 ffffffc0 00000000 00000000 fe2a8000 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.364656] e7e0  00000001 ffffffff 00000000 00000000 ffffffff 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.372953] e800  012951f0 ffffffc0 fe22b8d8 ffffffc0 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.381238] e820  80000000 00000000 ffefffff 00000000 00000001 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.389532] e840  fe21bb40 ffffffc0 fe2767c0 ffffffc0 00000000 00000000 fe2b0000 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.397830] e860  00000001 ffffffff 00000000 00000000 ffffffff 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.406132]
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.477532] X26: 0xffffffc0a823ff80:
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.482576] ff80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.490861] ffa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.499146] ffc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.507444] ffe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.515731] 0000  00000000 00000000 ffffffff ffffffff ee509380 ffffffc0 000bb27c ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.524022] 0020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.532316] 0040  00000000 00000000 00000003 00000000 00000100 00000000 57ac6e9d 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.540610] 0060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.548901]
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.548901] X29: 0xffffffc0a8243d60:
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.553944] 3d60  01179758 ffffffc0 caa45898 ffffffc0 a48b0000 ffffffc0 a48b1800 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.562245] 3d80  000080d0 00000000 00000400 00000000 00000420 00000000 000000ff 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.570539] 3da0  00097f80 ffffffc0 a61af000 ffffffc0 d9dfaa00 ffffffc0 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.578832] 3dc0  00002000 00000002 00000000 00010000 fcfc4844 ffffffbf 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.587134] 3de0  d8672e40 0000007f 001b4820 ffffffc0 b0c3a000 0000007f ea082860 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.595425] 3e00  00000003 00000000 00000003 00000000 b0c3a000 0000007f 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.603715] 3e20  00000116 00000000 0000001d 00000000 01178000 ffffffc0 a8240000 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.612008] 3e40  001b5418 ffffffc0 ffffffff ffffffff b4ed4ec0 ffffffc0 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.620311]
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.621801] Process ql2test (pid: 1854, stack limit = 0xffffffc0a8240058)
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.628581] Stack: (0xffffffc0a8243b80 to 0xffffffc0a8244000)
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.634321] 3b80: 00000000 00000000 fe26e800 ffffffc0 83fe0000 00000000 d9dfaa00 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.642485] 3ba0: 83fe0000 00000000 d9dfaa00 ffffffc0 00000000 00000000 83fe0000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.650651] 3bc0: 00000010 00000000 ca97c3f0 ffffffc0 008499d0 ffffffc0 1fe31670 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.658816] 3be0: ee5098a0 ffffffc0 83fe0000 00000000 01300900 ffffffc0 fe1983e8 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.666982] 3c00: fe2a8854 ffffffc0 00000010 00000000 00000010 00000000 000003f0 000003e0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.675145] 3c20: 040561c8 ffffffbc fe1983f4 ffffffc0 00000340 00000000 00000010 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.683311] 3c40: 00000200 00000000 fe273000 ffffffc0 fe198018 ffffffc0 83fe0000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.691476] 3c60: 00000000 00000000 00000010 00000000 fe22b940 ffffffc0 fe26e800 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.699640] 3c80: 00096e70 ffffffc0 a8243de0 ffffffc0 00001fff 00000000 00005400 00000001
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.707806] 3ca0: fe22b940 ffffffc0 00000010 00000000 00000010 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.715970] 3cc0: 000970b4 ffffffc0 00097018 ffffffc0 d9dfabe0 ffffffc0 00010000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.724136] 3ce0: 00000000 00000000 ca97c3e0 ffffffc0 00002000 00000000 a48b1800 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.732300] 3d00: 000001ff 00000000 caa45898 ffffffc0 00002000 00000000 ffffffff ffffffff
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.740467] 3d20: 00097e30 ffffffc0 00000340 00000000 a8243de0 ffffffc0 a8243de0 00000002
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.748631] 3d40: d9dfaa00 ffffffc0 fe22b970 ffffffc0 cbe693c0 ffffffc0 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.756795] 3d60: 01179758 ffffffc0 caa45898 ffffffc0 a48b0000 ffffffc0 a48b1800 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.764960] 3d80: 000080d0 00000000 00000400 00000000 00000420 00000000 000000ff 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.773125] 3da0: 00097f80 ffffffc0 a61af000 ffffffc0 d9dfaa00 ffffffc0 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.781291] 3dc0: 00002000 00000002 00000000 00010000 fcfc4844 ffffffbf 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.789455] 3de0: d8672e40 0000007f 001b4820 ffffffc0 b0c3a000 0000007f ea082860 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.797619] 3e00: 00000003 00000000 00000003 00000000 b0c3a000 0000007f 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.805782] 3e20: 00000116 00000000 0000001d 00000000 01178000 ffffffc0 a8240000 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.813947] 3e40: 001b5418 ffffffc0 ffffffff ffffffff b4ed4ec0 ffffffc0 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.822112] 3e60: 001b54fc ffffffc0 00000000 00000000 001b54e0 ffffffc0 001b54c8 ffffffc0
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.830278] 3e80: 00401510 00000000 00000000 00000000 ffffffff ffffffff b2da70cc 0000007f
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.838444] 3ea0: 80000000 00000000 00000015 00000000 000853c8 ffffffc0 00000015 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.846609] 3ec0: 000853c8 ffffffc0 00000000 00000000 00000003 00000000 00000003 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.854774] 3ee0: b0c3a000 0000007f 00000ff0 00000000 b0c3a000 0000007f 00000002 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.862940] 3f00: b2e290d0 0000007f 00000000 00000000 0000001d 00000000 9b9a9998 9f9e9d9c
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.871105] 3f20: 01010101 01010101 00000028 00000000 b3b2b1b0 b7b6b5b4 bbbab9b8 bfbebdbc
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.879271] 3f40: 00000000 00000000 b302acc0 0000007f b2da70c0 0000007f 00412070 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.887436] 3f60: 00000a03 00000000 00401510 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.895602] 3f80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.903767] 3fa0: 00000000 00000000 00000000 00000000 00000000 00000000 d8672e40 0000007f
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.911930] 3fc0: 004012c0 00000000 d8672e40 0000007f b2da70cc 0000007f 80000000 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.920099] 3fe0: 00000003 00000000 0000001d 00000000 000001d0 00000000 00000200 00000000
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.928261] Call trace:
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.930706] [<ffffffc00084dfac>] smmu_iommu_map_sg+0x2ec/0x408
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.936533] Code: aa1c03e0 97eb4c4a aa0003fc b4000200 (f9400018)
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.942781] ---[ end trace b543f9f91a9355b9 ]---
Nov 24 15:41:02 tegra-ubuntu kernel: [   68.949202] note: ql2test[1854] exited with preempt_count 1

vidyas · December 2, 2016, 1:46pm

Can you give us more information on version? Ex? 23.2 / 24.2 / 24.2 Etc…?

mrjazz2 · December 6, 2016, 3:49pm

This problem exists with 24.2 as well as with 24.2.1. What do you mean with “Ex”?
By the way, besides this issue there is also the problem that multiple base register accesses with iowrite32 (and related functions) eventually lead to a stalling system. At least pure DMA-activity (if the dma-map function didn’t cause problems) seems to work for a longer period of time without crashing.

vidyas · December 16, 2016, 9:02am

would you be able to share with us the disassembled code of smmu_iommu_map_sg() API?
‘$CROSS_COMPILE-objdump -D’ on tegra-smmu.o would give that output.
We tried to do the same on our local build but the offsets don’t seem to be matching.
We are checking with our memory/SMMU team also in parallel.

mrjazz2 · December 17, 2016, 1:57pm

In this file there is the disassembly of tegra-smmu.o as well as the accordant kernel.log-excerpt of the error (might differ from the upper):
http://dominik.ws/smmu-problem.zip

vidyas · December 18, 2016, 6:16pm

Thanks for sharing the files.
It looks like the issue comes when ‘(sg)-page_link’ gets executed. So, one of the sg entries in the list passed to smmu_iommu_map_sg() API has an invalid pointer.
Does this issue occur every time? if not, how often do you encounter this issue?

mrjazz2 · December 18, 2016, 10:51pm

This issue occurs about every third time, but it is random. This is the relevant code from my kernel-module:

#define QL2_USERPTR_PAGE_COUNT (1<<(26-PAGE_SHIFT))
#define QL2_USERPTR_PAGES_PER_SGLIST (1<<5)
#define QL2_USERPTR_SGLIST_COUNT (QL2_USERPTR_PAGE_COUNT/QL2_USERPTR_PAGES_PER_SGLIST)

userPages = (struct page**)kzalloc(sizeof(struct page*)*QL2_USERPTR_PAGE_COUNT, GFP_KERNEL);
if (!userPages) {
    printk(KERN_ERR "ql2: Could not alloc userPages-array.\n");
    return -ENOMEM;
}
down_read(&current->mm->mmap_sem);
count = get_user_pages(current, current->mm, ioctl_param, QL2_USERPTR_PAGE_COUNT, 1, 1, userPages, NULL);
up_read(&current->mm->mmap_sem);
if (count != QL2_USERPTR_PAGE_COUNT) {
    printk(KERN_ERR "ql2: userPagesCount invalid (%d != %d).\n", count, QL2_USERPTR_PAGE_COUNT);
    return -ENOMEM;
}
sglistlist = kzalloc(sizeof(struct scatterlist*)*QL2_USERPTR_SGLIST_COUNT, GFP_KERNEL);
if (!sglistlist) {
    printk(KERN_ERR "ql2: Could not alloc sglistlist.\n");
    return -ENOMEM;
}
for (i = 0; i < QL2_USERPTR_SGLIST_COUNT; ++i) {
    sglistlist[i] = kzalloc(sizeof(struct scatterlist)*(QL2_USERPTR_PAGES_PER_SGLIST + (i==(QL2_USERPTR_SGLIST_COUNT-1)?0:1)), GFP_KERNEL);
    for (k = 0; k < QL2_USERPTR_PAGES_PER_SGLIST; ++k) {
        struct page *p = userPages[k+i*QL2_USERPTR_PAGES_PER_SGLIST];
        sg_set_page(&sglistlist[i][k], p, PAGE_SIZE, 0); 
    }
    if (i > 0) sg_chain(sglistlist[i-1], QL2_USERPTR_PAGES_PER_SGLIST+1, sglistlist[i]);
}
printk(KERN_INFO "ql2: Attempting to call pci_map_sg.\n");
pci_map_sg(pdev, sglistlist[0], QL2_USERPTR_PAGE_COUNT, PCI_DMA_FROMDEVICE);

“ioctl_param” is the pointer to the user-space memory. The memory is allocated in this way:

void *memHost;
cudaHostAlloc(&memHost, 1<<26, cudaHostAllocMapped);

Do you have any idea what could be the problem?

vidyas · December 21, 2016, 11:40am

I think you need to make the following change in your code i.e. you should mark last sg entry as the ‘end’ exclusively with sg_mark_end() API.

for (k = 0; k < QL2_USERPTR_PAGES_PER_SGLIST; ++k) {
                struct page *p = pd->userPages[k+i*QL2_USERPTR_PAGES_PER_SGLIST];
                sg_set_page(&sglistlist[i][k], p, PAGE_SIZE, 0);
+               if ((i == (QL2_USERPTR_SGLIST_COUNT - 1)) &&
+                   (k == (QL2_USERPTR_PAGES_PER_SGLIST - 1)))
+                   sg_mark_end(&sglistlist[i][k]);
            }
            if (i > 0) {
                sg_chain(sglistlist[i-1], QL2_USERPTR_PAGES_PER_SGLIST+1, sglistlist[i]);
            }

Can you please try with above change and let us know if it works.

mrjazz2 · December 21, 2016, 2:33pm

I think this solved my problem. I tried it by now many times and the error didn’t occur a single time since I marked the end with sg_mark_end. Thank you!

Topic		Replies	Views
Stability issue with DMA-memory transfer while GPU activity Jetson TX1	17	4110	October 18, 2021
Serial-Tegra DMA Driver Bug - memory leak Jetson Orin NX kernel	8	298	June 17, 2024
Jetson crashes Jetson TX2	10	1199	August 11, 2023
TX2 unexcept shutdown caused by “Failed to set dvfs regulator vdd-core” Jetson TX2	4	476	October 18, 2021
all usb devices got disconected Jetson TX2	3	764	October 18, 2021
Jetson TK1 won't boot Jetson TK1	9	2563	October 18, 2021
Jetson TX2 rebooting due to corrupted rootfs - any pointers to recover it? Jetson TX2	19	5218	October 18, 2021
Jetson Xavier NX, kernel panic at pc : tegra194_cbb_err_isr+0x19c/0x1b0 Jetson Xavier NX kernel	2	202	May 15, 2024
Failed to boot TX2 Jetson TX2	3	872	October 18, 2021
PCIe serial card problem Jetson TX1	12	2377	October 18, 2021

pci_map_sg sometimes leads to kernel-oops

Related topics