Summary
A single thread runs this loop on a compute-capable queue (family 0, queue 0), waiting for each fence before going on:
- Create a new image and its memory, and fill it with
vkCmdCopyImage. - Create a view of it and write it into a descriptor set.
- Destroy the previous frame’s view, image and memory. Their last use had completed, since its fence was waited on.
- Dispatch a compute shader that reads the new image through that descriptor.
The bug needs other threads calling vkQueueSubmit2 on other VkQueues at the same time. Those submits can be empty, can come from any queue family, and need no semaphores. When they run, some dispatches read the image bound by the previous dispatch instead of the one in the current descriptor:
- Previous image still alive: the dispatch silently reads the previous frame’s content, on 0.5 % to 24 % of frames.
- Previous image freed: the read hits an unmapped VA, giving an MMU fault, Xid 109 (CTX SWITCH TIMEOUT) and
VK_ERROR_DEVICE_LOST, within about 255 frames.
Without the concurrent submits, nothing goes wrong over 93,000 frames. Vulkan validation, including synchronization validation, reports no error.
We first saw it in production. There, a video decode queue submits while a compute shader samples NV12 frames through a VkSamplerYcbcrConversion. The attached program has no video decoding, and the bug also shows without Y’CbCr.
Smallest reproducer
repro.cc (350 lines, Vulkan 1.1, no extensions) and repro.comp show the fault in about a second:
glslc -fshader-stage=comp repro.comp -o repro.spv
g++ -O2 -std=c++17 -o repro repro.cc -lvulkan -lpthread
./repro
It creates two queues in family 0. The main thread does the following per frame on queue 0:
- Create a 4096×2048
VK_FORMAT_G8_B8R8_2PLANE_420_UNORMimage (MUTABLE_FORMAT | EXTENDED_USAGE) and anR8_UNORMview of plane 0. - Destroy the previous frame’s image.
- In one command buffer,
vkCmdCopyImageplane 0 from a source image filled with 0x00 or 0xFF, alternating each frame, then dispatch a shader thattexelFetches every texel and counts the ones that differ from the expected value. - Submit, then wait for the fence.
A second thread submits zero command buffers to queue 1 with its own fence, timed so that the submit lands just as the main thread submits.
nvidia-t4-stale-image-repro.zip (393.8 KB)