Imported CPU-cached dma-buf mapped non-coherent: GPU reads stale data (EGL dma-buf import, 615.71.09)

Graphics / Linux Linux
1

Summary

When a dma-buf backed by ordinary write-back cached system memory (udmabuf, or any similar exporter) is imported through EGL_EXT_image_dma_buf_import and sampled, the GPU reads stale data. The data is stale even when the CPU writes are complete and fenced, and even when they are bracketed with DMA_BUF_IOCTL_SYNC. The GPU reads the correct data only if the CPU explicitly writes its caches back with CLFLUSH first, or if PCIe No Snoop Enable is cleared on the GPU. Toggling that single bit switches the failure from 200/200 iterations to 0/200.

This looks like the import being mapped with the non-coherent system-memory aperture (No Snoop TLPs) even though RM knows the pages are CPU-cached. On x86, dma_sync_*_for_device() is a no-op, so an exporter or a CPU writer has no API to make this work. The importer has to snoop.

Real-world impact: Looking Glass (KVMFR dma-buf, vmalloc_user() pages) shows corrupted window edges on RTX hosts when DMA import is enabled. The corruption disappears when No Snoop Enable is cleared.

System

  • GPU: NVIDIA GeForce RTX 3070 Ti (GA104), PCIe, used for desktop
  • Driver: 615.71.09, open kernel modules (CachyOS linux-cachyos-bore-nvidia-open)
  • Kernel: 7.2.8-1-cachyos-bore, intel_iommu=on iommu=pt (identity domain for the GPU)
  • CPU: Intel Raptor Lake-S (family 6, model 183, stepping 1, microcode 0x137; engineering/QS sample), MSI MAG Z690 TOMAHAWK WIFI DDR4
  • Compositor: Hyprland 0.56.2 (Wayland); the repro is headless (EGL_PLATFORM_DEVICE_EXT)

Reproduction

Self-contained repro, about 300 lines of C, no external project needed. It is attached as nv-dmabuf-snoop-repro.zip (source, matrix script, full results), and the source is also included at the end of this post.

cc -O2 -Wall -o nv-dmabuf-snoop nv-dmabuf-snoop.c $(pkg-config --cflags libdrm) -lEGL -lGLESv2
# needs read/write access to /dev/udmabuf
./nv-dmabuf-snoop 200 1024 256 each p 1 1   # plain: CPU writes, no flush
./nv-dmabuf-snoop 200 1024 256 each f 1 1   # control: same, plus CLFLUSH before GPU read

What it does:

  1. Creates a memfd, exports it through /dev/udmabuf (cached shmem pages), and imports it as a linear DRM_FORMAT_ABGR8888 EGLImage → GL_TEXTURE_EXTERNAL_OES.
  2. Each iteration, the CPU writes a new pattern into every pixel with plain stores. With sync=1 it writes through the dma-buf mmap, bracketed by DMA_BUF_IOCTL_SYNC START/END. Then MFENCE.
  3. A blit of a 16 MiB VRAM surface evicts the GPU L2 (evict=1). The external texture is re-bound, sampled with texelFetch into an FBO, and read back with glReadPixels.
  4. Every pixel is compared with what the CPU wrote.
  5. The only difference between modes p and f is a CLFLUSH over the buffer before the GPU read.

A sanity check verifies that the initially flushed frame reads back exactly before the loop starts.

Results (200 iterations each, 262144 pixels per iteration)

All rows use the same binary. No Snoop Enable was cleared with setpci -s 01:00.0 CAP_EXP+8.w=0000:0800, which changes only the No Snoop Enable bit (DevCtl NoSnoop+ → NoSnoop-), and restored afterwards.

GPU L2 evicted before each read (evict=1). This isolates CPU-cache coherence:

mode DMA_BUF_IOCTL_SYNC No Snoop Enable set (default) No Snoop Enable cleared
plain (no CPU flush) no 200 / 200 bad (up to 240520 px) 0 / 200
plain (no CPU flush) yes 200 / 200 bad (up to 262048 px) 0 / 200
CLFLUSH before read no 0 / 200 0 / 200
CLFLUSH before read yes 0 / 200 0 / 200

No GPU L2 eviction (evict=0). This is a second, independent effect (see below):

mode DMA_BUF_IOCTL_SYNC No Snoop Enable set No Snoop Enable cleared
plain no 200 / 200 151 / 200
CLFLUSH before read no 153 / 200 142 / 200
plain yes 200 / 200 162 / 200
CLFLUSH before read yes 150 / 200 149 / 200

Full output is in nv-dmabuf-snoop-repro.zip (results-nosnoop-on.txt, results-nosnoop-off.txt).

Analysis

Kernel-side tracing during import, using kprobes on the open kernel module’s interface layer (RM-core symbols reject kprobes):

  • nv_register_sgt() receives cache_type = 0 (NV_MEMORY_CACHED) for the imported buffer, so RM knows the pages are CPU-cached.
  • The matching NV_ESC_RM_MAP_MEMORY_DMA ioctl from the EGL driver passes NVOS46_PARAMETERS.flags = 0x0, which is NVOS46_FLAGS_CACHE_SNOOP_DISABLE. It is issued from the same thread within ~0.3 ms, with a length equal to the imported buffer size.
    • In the same trace the driver passes CACHE_SNOOP_ENABLE (flags & 0x10) for some of its own allocations.
  • In the 615.71.09 open source, the import memdesc keeps cpuCacheSnoop = MEMDESC_CACHE_SNOOP_DEFER_TO_MAP:
    • It is zero-initialised in memdescCreate.
    • _createMemdescFromDmaBufSgtHelper() never sets it.
  • Because of that, dmaAllocMapping_GM107() (virt_mem_allocator_gm107.c, the non-fully-coherent branch) selects GMMU_APERTURE_SYS_NONCOH from the map flag.
  • Non-coherent aperture plus PCIe No Snoop Enable gives reads that bypass the CPU caches, which matches the measurements above.

Why the writer cannot fix it with the documented API: on x86, udmabuf’s end_cpu_access → dma_sync_sgtable_for_device() does nothing because x86 DMA is assumed to be cache-coherent. The sync=1 rows show DMA_BUF_IOCTL_SYNC makes no difference.

Expected behaviour

A foreign dma-buf whose pages RM has registered as NV_MEMORY_CACHED should be mapped with the coherent aperture (SYS_COH), so the GPU snoops. Alternatively the driver must write back CPU caches itself before GPU access. Exporters on x86 get no other signal.

Possible fix (untested suggestion)

Either the EGL/GL driver passes NVOS46_FLAGS_CACHE_SNOOP_ENABLE when mapping imported dma-bufs, or RM forces snooping for CPU-cached imports:

--- a/src/nvidia/arch/nvalloc/unix/src/osmemdesc.c
+++ b/src/nvidia/arch/nvalloc/unix/src/osmemdesc.c
@@ _createMemdescFromDmaBufSgtHelper
     pMemDesc = *ppMemDesc;

     memdescSetGpuCacheAttrib(pMemDesc, gpuCachedFlags);
+    // Pages of a foreign dma-buf may be dirty in CPU caches; on platforms
+    // where DMA is assumed coherent the exporter has no way to flush them.
+    if (cacheType == NV_MEMORY_CACHED && !isPeerMmio)
+        memdescSetCpuCacheSnoop(pMemDesc, MEMDESC_CACHE_SNOOP_ENABLE);
     memdescSetMemData(pMemDesc, NULL, NULL);

Second observation (possibly separate)

Without the GPU L2 eviction step (evict=0), about 70–80% of iterations return old data. This happens even with the CLFLUSH control and even with No Snoop Enable cleared (142–162 / 200), so it does not depend on CPU caches or PCIe snooping. Lines of the imported sysmem buffer appear to stay in the GPU L2 across submissions and external texture re-binds, so a consumer that re-binds after the producer has finished writing can still sample old contents. With a 1 MiB buffer the whole image fits in the 4 MiB L2. Large buffers (for example a 33 MB 4K frame) mostly hide this effect by evicting the L2 themselves.

Attachments

nv-dmabuf-snoop.c 
// nv-dmabuf-snoop: stale GPU reads from a CPU-cached dma-buf imported into
// NVIDIA EGL.
//
// A memfd is exported through /dev/udmabuf (ordinary write-back cached shmem
// pages), imported with EGL_EXT_image_dma_buf_import as a linear
// GL_TEXTURE_EXTERNAL_OES and sampled into an FBO that is read back.
// Every iteration the CPU rewrites the whole buffer with a new pattern using
// ordinary (cacheable, non-temporal-free) stores, then the GPU samples it.
//
// Modes, interleaved so they share the same thermal/cache conditions:
//   plain  - no flush: dirty lines may still sit in the CPU caches
//   delay  - busy-wait as long as the flush takes, but no flush (timing control)
//   flush  - CLFLUSH every line + MFENCE before the GPU reads (coherence control)
//
// If the importer accesses this memory with snooped (coherent) reads, every
// mode reads back exactly what the CPU wrote. Mismatches only in plain/delay
// that disappear with flush mean the GPU read DRAM while newer data was in
// the CPU cache, i.e. non-snooped access to cacheable memory.
//
// Build: cc -O2 -Wall -o nv-dmabuf-snoop nv-dmabuf-snoop.c
//          $(pkg-config --cflags libdrm) -lEGL -lGLESv2   (one line)
// Run:   ./nv-dmabuf-snoop [iterations=300] [width=1024] [height=256] [bind=each|once] [modes=pdf] [evict=0|1] [sync=0|1]
//
// evict=1 blits a 16 MiB VRAM surface before each sample so lines fetched
// by earlier iterations cannot be served from the GPU L2.
//
// sync=1 writes through the dma-buf's own mmap and brackets every frame with
// DMA_BUF_IOCTL_SYNC START/END (WRITE), the documented CPU-access protocol.
// sync=0 writes through the memfd mapping (same pages).
//
// modes selects which of p(lain), d(elay), f(lush) are cycled. Run "f" alone
// as the clean control: interleaving lets stale lines fetched by a plain run
// survive in GPU-side caches and pollute the following flush run.
//
// bind=each (default) re-binds the external texture before every draw, as
// real compositors/clients do; the NVIDIA driver latches external texture
// contents at bind time, so bind=once samples an old copy regardless of
// cache coherence and is only a separate observation.

#define _GNU_SOURCE
#include <EGL/egl.h>
#include <EGL/eglext.h>
#include <GLES3/gl3.h>
#include <GLES2/gl2ext.h>
#include <drm_fourcc.h>
#include <emmintrin.h>
#include <fcntl.h>
#include <linux/dma-buf.h>
#include <linux/udmabuf.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <time.h>
#include <unistd.h>

#define CACHE_LINE 64

enum Mode { MODE_PLAIN, MODE_DELAY, MODE_FLUSH, MODE_COUNT };
static const char * modeName[MODE_COUNT] = { "plain", "delay", "flush" };

struct Stats
{
  unsigned runs, dirtyRuns;
  size_t badPixels, stalePixels, maxBadPixels;
};

static void die(const char * what)
{
  fprintf(stderr, "error: %s\n", what);
  exit(1);
}

static double now(void)
{
  struct timespec ts;
  clock_gettime(CLOCK_MONOTONIC, &ts);
  return ts.tv_sec + ts.tv_nsec * 1e-9;
}

// The pattern changes every pixel between consecutive iterations, so a
// mismatching pixel that equals the previous iteration's value is stale data.
static inline uint32_t pattern(unsigned iter, size_t pixel)
{
  uint8_t r = (uint8_t)(iter * 37 + 11);
  uint8_t g = (uint8_t)(iter * 101 + 3);
  uint8_t b = (uint8_t)(pixel ^ (pixel >> 8));
  return (uint32_t)r | (uint32_t)g << 8 | (uint32_t)b << 16 | 0xffu << 24;
}

static void write_frame(volatile uint32_t * dst, size_t pixels, unsigned iter)
{
  // volatile keeps these as plain stores; libc memcpy may switch to
  // non-temporal stores for large sizes and hide the effect.
  for (size_t i = 0; i < pixels; ++i)
    dst[i] = pattern(iter, i);
}

static void flush_frame(const void * p, size_t bytes)
{
  _mm_mfence();
  for (size_t off = 0; off < bytes; off += CACHE_LINE)
    _mm_clflush((const char *)p + off);
  _mm_mfence();
}

static GLuint compile(GLenum type, const char * src)
{
  GLuint s = glCreateShader(type);
  glShaderSource(s, 1, &src, NULL);
  glCompileShader(s);
  GLint ok;
  glGetShaderiv(s, GL_COMPILE_STATUS, &ok);
  if (!ok)
  {
    char log[1024];
    glGetShaderInfoLog(s, sizeof(log), NULL, log);
    fprintf(stderr, "shader: %s\n", log);
    die("shader compile");
  }
  return s;
}

static EGLDisplay open_nvidia_display(void)
{
  PFNEGLQUERYDEVICESEXTPROC queryDevices =
    (PFNEGLQUERYDEVICESEXTPROC)eglGetProcAddress("eglQueryDevicesEXT");
  PFNEGLGETPLATFORMDISPLAYEXTPROC platformDisplay =
    (PFNEGLGETPLATFORMDISPLAYEXTPROC)eglGetProcAddress("eglGetPlatformDisplayEXT");
  if (!queryDevices || !platformDisplay)
    die("EGL_EXT_device_enumeration / platform_device missing");

  EGLDeviceEXT devices[16];
  EGLint count = 0;
  queryDevices(16, devices, &count);
  for (EGLint i = 0; i < count; ++i)
  {
    EGLDisplay d = platformDisplay(EGL_PLATFORM_DEVICE_EXT, devices[i], NULL);
    if (d == EGL_NO_DISPLAY || !eglInitialize(d, NULL, NULL))
      continue;
    const char * vendor = eglQueryString(d, EGL_VENDOR);
    if (vendor && strstr(vendor, "NVIDIA"))
      return d;
    eglTerminate(d);
  }
  die("no NVIDIA EGL device");
  return EGL_NO_DISPLAY;
}

int main(int argc, char ** argv)
{
  const unsigned iterations = argc > 1 ? (unsigned)atoi(argv[1]) : 300;
  const unsigned width      = argc > 2 ? (unsigned)atoi(argv[2]) : 1024;
  const unsigned height     = argc > 3 ? (unsigned)atoi(argv[3]) : 256;
  const bool     bindEach   = !(argc > 4 && !strcmp(argv[4], "once"));
  const char *   modes      = argc > 5 ? argv[5] : "pdf";
  const bool     evict      = argc > 6 && atoi(argv[6]);
  const bool     useSync    = argc > 7 && atoi(argv[7]);
  enum Mode cycle[MODE_COUNT];
  unsigned cycleLength = 0;
  for (const char * c = modes; *c && cycleLength < MODE_COUNT; ++c)
    if (*c == 'p') cycle[cycleLength++] = MODE_PLAIN;
    else if (*c == 'd') cycle[cycleLength++] = MODE_DELAY;
    else if (*c == 'f') cycle[cycleLength++] = MODE_FLUSH;
  if (!cycleLength)
    die("modes must contain p, d and/or f");
  const size_t   pitch      = (size_t)width * 4;
  const size_t   pixels     = (size_t)width * height;
  const long     page       = sysconf(_SC_PAGESIZE);
  const size_t   size       = (pitch * height + page - 1) / page * page;

  int memfd = memfd_create("nv-dmabuf-snoop", MFD_ALLOW_SEALING);
  if (memfd < 0 || ftruncate(memfd, size) < 0 ||
      fcntl(memfd, F_ADD_SEALS, F_SEAL_SHRINK) < 0)
    die("memfd");
  uint32_t * cpu = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, memfd, 0);
  if (cpu == MAP_FAILED)
    die("mmap memfd");

  int udev = open("/dev/udmabuf", O_RDWR | O_CLOEXEC);
  if (udev < 0)
    die("open /dev/udmabuf (need access, e.g. kvm group)");
  struct udmabuf_create create = { .memfd = (uint32_t)memfd,
    .flags = UDMABUF_FLAGS_CLOEXEC, .offset = 0, .size = size };
  int dmabuf = ioctl(udev, UDMABUF_CREATE, &create);
  close(udev);
  if (dmabuf < 0)
    die("UDMABUF_CREATE");

  if (useSync)
  {
    cpu = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, dmabuf, 0);
    if (cpu == MAP_FAILED)
      die("mmap dma-buf");
  }

  EGLDisplay display = open_nvidia_display();
  if (!eglBindAPI(EGL_OPENGL_ES_API))
    die("eglBindAPI");
  EGLint configAttrs[] = { EGL_RENDERABLE_TYPE, EGL_OPENGL_ES3_BIT,
    EGL_SURFACE_TYPE, EGL_PBUFFER_BIT, EGL_NONE };
  EGLConfig config;
  EGLint configs;
  if (!eglChooseConfig(display, configAttrs, &config, 1, &configs) || configs < 1)
    die("eglChooseConfig");
  EGLint contextAttrs[] = { EGL_CONTEXT_CLIENT_VERSION, 3, EGL_NONE };
  EGLContext context = eglCreateContext(display, config, EGL_NO_CONTEXT, contextAttrs);
  if (context == EGL_NO_CONTEXT ||
      !eglMakeCurrent(display, EGL_NO_SURFACE, EGL_NO_SURFACE, context))
    die("EGL context");

  PFNEGLCREATEIMAGEKHRPROC createImage =
    (PFNEGLCREATEIMAGEKHRPROC)eglGetProcAddress("eglCreateImageKHR");
  PFNGLEGLIMAGETARGETTEXTURE2DOESPROC imageTarget =
    (PFNGLEGLIMAGETARGETTEXTURE2DOESPROC)eglGetProcAddress("glEGLImageTargetTexture2DOES");
  if (!createImage || !imageTarget)
    die("EGLImage entry points");

  // Fill once and make sure DRAM holds a known state before the first import.
  write_frame(cpu, pixels, 0);
  flush_frame(cpu, size);

  EGLint imageAttrs[] = { EGL_WIDTH, (EGLint)width, EGL_HEIGHT, (EGLint)height,
    EGL_LINUX_DRM_FOURCC_EXT, DRM_FORMAT_ABGR8888,
    EGL_DMA_BUF_PLANE0_FD_EXT, dmabuf,
    EGL_DMA_BUF_PLANE0_OFFSET_EXT, 0,
    EGL_DMA_BUF_PLANE0_PITCH_EXT, (EGLint)pitch,
    EGL_DMA_BUF_PLANE0_MODIFIER_LO_EXT, (EGLint)(DRM_FORMAT_MOD_LINEAR & 0xffffffff),
    EGL_DMA_BUF_PLANE0_MODIFIER_HI_EXT, (EGLint)(DRM_FORMAT_MOD_LINEAR >> 32),
    EGL_NONE };
  EGLImageKHR image = createImage(display, EGL_NO_CONTEXT,
    EGL_LINUX_DMA_BUF_EXT, NULL, imageAttrs);
  if (image == EGL_NO_IMAGE_KHR)
  {
    fprintf(stderr, "eglCreateImageKHR error 0x%x\n", eglGetError());
    die("dma-buf import");
  }

  GLuint texture;
  glGenTextures(1, &texture);
  glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
  glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
  glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
  imageTarget(GL_TEXTURE_EXTERNAL_OES, image);

  GLuint target, fbo;
  glGenTextures(1, &target);
  glBindTexture(GL_TEXTURE_2D, target);
  glTexStorage2D(GL_TEXTURE_2D, 1, GL_RGBA8, width, height);
  glGenFramebuffers(1, &fbo);
  glBindFramebuffer(GL_FRAMEBUFFER, fbo);
  glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, target, 0);
  if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE)
    die("FBO incomplete");

  const char * vs = "#version 300 es\n"
    "void main(){ vec2 p = vec2(float((gl_VertexID << 1) & 2), float(gl_VertexID & 2));"
    " gl_Position = vec4(p * 2.0 - 1.0, 0.0, 1.0); }\n";
  const char * fs = "#version 300 es\n"
    "#extension GL_OES_EGL_image_external_essl3 : require\n"
    "precision highp float; uniform samplerExternalOES src; out vec4 color;\n"
    "void main(){ color = texelFetch(src, ivec2(gl_FragCoord.xy), 0); }\n";
  GLuint prog = glCreateProgram();
  glAttachShader(prog, compile(GL_VERTEX_SHADER, vs));
  glAttachShader(prog, compile(GL_FRAGMENT_SHADER, fs));
  glLinkProgram(prog);
  GLint linked;
  glGetProgramiv(prog, GL_LINK_STATUS, &linked);
  if (!linked)
    die("program link");
  glUseProgram(prog);
  glUniform1i(glGetUniformLocation(prog, "src"), 0);
  glActiveTexture(GL_TEXTURE0);
  glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
  glViewport(0, 0, width, height);
  glDisable(GL_BLEND);
  glDisable(GL_DITHER);
  glPixelStorei(GL_PACK_ALIGNMENT, 1);

  // Two 2048x2048 RGBA8 surfaces: one blit reads 16 MiB and writes 16 MiB,
  // several times the RTX 3070 Ti L2 (4 MiB).
  GLuint evictTex[2], evictFbo[2];
  const GLint evictSize = 2048;
  glGenTextures(2, evictTex);
  glGenFramebuffers(2, evictFbo);
  for (int i = 0; i < 2; ++i)
  {
    glBindTexture(GL_TEXTURE_2D, evictTex[i]);
    glTexStorage2D(GL_TEXTURE_2D, 1, GL_RGBA8, evictSize, evictSize);
    glBindFramebuffer(GL_FRAMEBUFFER, evictFbo[i]);
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, evictTex[i], 0);
    glClearColor(i, 0, 0, 1);
    glClear(GL_COLOR_BUFFER_BIT);
  }
  glBindFramebuffer(GL_FRAMEBUFFER, fbo);
  glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);

  uint32_t * readback = malloc(pixels * 4);
  if (!readback)
    die("malloc");

  // Sanity: the flushed initial frame must read back exactly, otherwise the
  // import/shader path itself is wrong and nothing below means anything.
  glDrawArrays(GL_TRIANGLES, 0, 3);
  glReadPixels(0, 0, width, height, GL_RGBA, GL_UNSIGNED_BYTE, readback);
  if (glGetError() != GL_NO_ERROR)
    die("GL error in sanity draw");
  for (size_t i = 0; i < pixels; ++i)
    if (readback[i] != pattern(0, i))
    {
      fprintf(stderr, "sanity mismatch at pixel %zu: got %08x want %08x\n",
        i, readback[i], pattern(0, i));
      die("sanity check failed; import or sampling path is broken");
    }

  // Measure one flush so the delay control waits a comparable time.
  double t0 = now();
  flush_frame(cpu, size);
  const double flushTime = now() - t0;

  printf("GL_RENDERER: %s\nGL_VERSION:  %s\n", glGetString(GL_RENDERER),
    glGetString(GL_VERSION));
  printf("buffer: %ux%u ABGR8888 linear, %zu bytes, udmabuf; flush takes %.1f us; bind=%s modes=%s evict=%d sync=%d\n",
    width, height, size, flushTime * 1e6, bindEach ? "each" : "once", modes, evict, useSync);

  struct Stats stats[MODE_COUNT] = { 0 };
  for (unsigned iter = 1; iter <= iterations; ++iter)
  {
    const enum Mode mode = cycle[iter % cycleLength];
    if (useSync)
    {
      struct dma_buf_sync sync = { DMA_BUF_SYNC_START | DMA_BUF_SYNC_WRITE };
      if (ioctl(dmabuf, DMA

_BUF_IOCTL_SYNC, &sync) < 0)
        die("DMA_BUF_IOCTL_SYNC start");
    }
    write_frame(cpu, pixels, iter);
    if (useSync)
    {
      struct dma_buf_sync sync = { DMA_BUF_SYNC_END | DMA_BUF_SYNC_WRITE };
      if (ioctl(dmabuf, DMA_BUF_IOCTL_SYNC, &sync) < 0)
        die("DMA_BUF_IOCTL_SYNC end");
    }
    if (mode == MODE_FLUSH)
      flush_frame(cpu, size);
    else
    {
      _mm_mfence();
      if (mode == MODE_DELAY)
        for (double end = now() + flushTime; now() < end; )
          _mm_pause();
    }

    if (evict)
    {
      glBindFramebuffer(GL_READ_FRAMEBUFFER, evictFbo[iter & 1]);
      glBindFramebuffer(GL_DRAW_FRAMEBUFFER, evictFbo[~iter & 1]);
      glBlitFramebuffer(0, 0, evictSize, evictSize, 0, 0, evictSize, evictSize,
        GL_COLOR_BUFFER_BIT, GL_NEAREST);
      glBindFramebuffer(GL_FRAMEBUFFER, fbo);
    }
    if (bindEach)
      glBindTexture(GL_TEXTURE_EXTERNAL_OES, texture);
    glDrawArrays(GL_TRIANGLES, 0, 3);
    glReadPixels(0, 0, width, height, GL_RGBA, GL_UNSIGNED_BYTE, readback);
    if (glGetError() != GL_NO_ERROR)
      die("GL error in loop");

    size_t bad = 0, stale = 0;
    for (size_t i = 0; i < pixels; ++i)
      if (readback[i] != pattern(iter, i))
      {
        ++bad;
        if (readback[i] == pattern(iter - 1, i))
          ++stale;
      }

    struct Stats * s = &stats[mode];
    ++s->runs;
    if (bad)
    {
      ++s->dirtyRuns;
      s->badPixels += bad;
      s->stalePixels += stale;
      if (bad > s->maxBadPixels)
        s->maxBadPixels = bad;
    }
  }

  printf("\n%-6s %6s %10s %12s %12s %10s\n",
    "mode", "runs", "bad runs", "bad pixels", "stale (prev)", "max/run");
  for (int m = 0; m < MODE_COUNT; ++m)
    if (stats[m].runs)
    printf("%-6s %6u %10u %12zu %12zu %10zu\n", modeName[m], stats[m].runs,
      stats[m].dirtyRuns, stats[m].badPixels, stats[m].stalePixels,
      stats[m].maxBadPixels);

  const bool ranPlain = stats[MODE_PLAIN].runs || stats[MODE_DELAY].runs;
  const bool ranFlush = stats[MODE_FLUSH].runs;
  const bool plainBad = stats[MODE_PLAIN].dirtyRuns || stats[MODE_DELAY].dirtyRuns;
  const bool flushBad = stats[MODE_FLUSH].dirtyRuns;
  printf("\nverdict: ");
  if (ranFlush && flushBad)
    printf("mismatches even after CLFLUSH (not explained by CPU cache coherence alone)\n");
  else if (ranPlain && ranFlush)
    printf(plainBad ? "STALE READS without CLFLUSH, clean with CLFLUSH "
                      "(GPU does not snoop CPU caches for this dma-buf)\n"
                    : "clean (coherent)\n");
  else
    printf("%s (single mode; compare with a separate flush-only run)\n",
      (plainBad || flushBad) ? "mismatches" : "clean");
  return plainBad && !flushBad ? 2 : flushBad ? 3 : 0;
}