Peer-to-peer DMA transfers bug under Intel Vt-d IOMMU virtualization

Graphics / Linux Linux
1

Hello,

I am trying to track the source of a bug that makes the use of multiple GTX1080’s impossible when I turn on the IOMMU in Linux 4.8 (using either standard iommu=on or iommu=pt for passthrough mode) on a X99 board.

The bug can be triggered by running any peer-to-peer memory transfer, for example running the CUDA 8.0 Samples code 1_Utilities/p2pBandwidthLatencyTest from the terminal triggers the problem: the video driver (and as a result the X server) crashes immediately, and after multiple Ctrl-C’s and waiting for tens of seconds the server eventually restarts and I am presented with a login prompt to X Windows…

The relevant kernel error messages are (thousands of these lines, just a snippet below:)

[   51.691440] DMAR: DRHD: handling fault status reg 2
[   51.691450] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.691457] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.691462] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.691465] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.691470] DMAR: DRHD: handling fault status reg 400
[   51.740674] DMAR: DRHD: handling fault status reg 402
[   51.740683] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.740688] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.740693] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set

Cleary the above suggest that the CUDA driver is attempting DMA at an address for which the corresponding iommu page table entry write flag is not set, presumably because the driver has not properly registered/requested access via the general dma_map() kernel interface (https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt)

Scouting the net reveals a bug registered (188271 – IOMMU DMAR fault with NVIDIA CUDA peer to peer) for exactly the same reason on a totally different hardware (Supermicro Dual socket board) using Pascal Titan-X’s, so same architecture cards as mine. Interestingly enough, the kernel error messages in this report claim unauthorized access of exactly the same memory address! (f8139000, in bold below) :

[16193.666976] DMAR: [DMA Write] Request device [82:00.0] fault addr <b>f8139000</b> [fault reason 05] PTE Write access is not set (edited)

So this looks like a red flag that somehow the indirection afforded by the iommu is bypassed and the driver is using hardcoded DMA addresses. Please note that the author of the bug report claims that seting iommu=igfx_off somehow solves this, but really igfx_off per se should be irrelevant here, as there are no intel integrated graphics in these systems. What instead happens is that most likely iommu=igfx_off as opposed to iommu=on just turns off iommu altogether, allowing the dma to succeed. This is exactly what happens on my system too. So in other words the bug report merely states that turning off the iommu allows peer-to-peer tranfers to work. Still his detailed log files should be very useful for an independent manifestation of the same issue.

I am using an ASRock X99 board (x99e-itx/ac) with latest firmware, intel i6800k, dual Asus GTX-1080s Founder’s Edition, 32GB ram and Ubuntu 16.10 with all updates applied (kernel 4.8.0-37) with latest driver 378.13. All earlier drivers exhibit same symptoms. I have uploaded nvidia-bug-report.log.gz together with dmesg.txt, lspci.txt and_usr_lib_xorg_Xorg.0.crash which is the ubuntu bug report file that contains every imaginable system detail together with a full stack trace and coredump of the crashed X server at:

2

I’m having the same problem with two GTX980s, except when I disable iommu instead of dma transfers succeeding my computer crashes and restarts. It looks like other people are having this problem, too, judging by this tensorflow bug report: How to disable peer to peer gpu memory accessing? · Issue #7810 · tensorflow/tensorflow · GitHub

3

Six months later, using kernel 4.12 and 384.69 drivers and the problem still persists. Should we just assume that Nvidia does not care/want to fix this driver bug unless we shell out for Quadro/Tesla boards? I.e. is it intentional on their side (so that gaming boards don’t undercut their more enterprise oriented products)?

4

Hi all, and 18 months later, using kernel 4.15 with latest driver 418, the problem still persists. I used X399 mobo and 4 1080ti for machine learning, but obviously, Cuda performs badly in p2pBandwidthLatencyTest until I set iommu=soft.

5

Please read this:
[url]https://devtalk.nvidia.com/default/topic/1047121/linux/simplep2p-example-and-multi-gpu-network-training-causes-system-freeze-and-err-in-nvidia-smi/post/5313582/#5313582[/url]
tl;dr this has to be set up by the bios or you’ll have to manually set this on the pci bus. Nothing the driver or kernel can handle.

6

potential solution:

  1. disable IOMMU(vt-d) in bios or via grub parameter intel_iommu=off.
    or
  2. hack vt-d driver to setup dma remapping table for P2P cards. (I think this shall be done by someone like Linux Kernel commitee, hardware-vendor, if there are more and more application scenarios where IOV and P2P must be used concurrently)