Driver ported from 4.9 to kernel 5.10 no longer run

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hello, nVidia Support,

We have a set of PCIe driver for our device, which works well in kernel 4.9 and JetPack 32.5; the driver works in user mode.

After we flashed the system to SDK 5.0.2(kernel 5.10), and after re-compiling the driver, the driver no longer runs, and during DMA stage, it reports this error:

[ 416.088490] nvidia_smmu_context_fault_bank: 21715 callbacks suppressed
[ 416.088512] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x1965c2000, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.093484] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x19662c500, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.097095] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x15a292600, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.097410] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x15a29b480, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.097764] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x15a2a55c0, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.099987] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x15a2c9940, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.103981] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x15a343400, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.108308] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x151c8a380, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.112671] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x151cf7d40, fsynr=0x40011, cbfrsynra=0x145b, cb=10
[ 416.118429] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x152a58b80, fsynr=0x40011, cbfrsynra=0x145b, cb=10

If someone can advice any suggestions, we would very much appreciate it.

Thank you.
Mei Guodong

3

Hello nVidia Support,

Could anyone please help on this issue?

Thanks.
Mei Guodong

4

Maybe check whether the EP/RP IOVA and BAR address are correctly assigned in the driver or not?

5

Hello WayneWWW,

Thanks for your reply.

Let me give a bit more details about the issue description:

Firstly, the driver runs from user mode(and not in kernel mode). The driver works normally to access/read/write the memory space that reflects to the on-board FPGA registers. this means the EP’s BAR address etc. are all allocated correctly.

The issues comes when I do a DMA from either device to host, or from host to device. I allocate a memory buffer in host CPU, this host memory buffer is translated to a so-called “physical address”, and pass to the FPGA device for DMA. The device DMA data to the host memory. the program crashes when the DMA happens.

Does it mean the physical address being passed to the FPGA device is not correct? Is there any guideline how the host memory being mapped to an address that a device can DMA data to that address?

by default, does your kernel 5.9 make use of IOMMU? Should I disable the IOMMU? Is there any side-effect to disable the IOMMU?

Thank you.
Mei Guodong