Description
After migrating my Vulkan renderer from classic descriptor sets to VK_EXT_descriptor_heap, I observed device loss during storage image texel accesses. The failures correlate with higher GPU virtual addresses for the image backing memory allocations. Moving the affected images’ memory to lower addresses restored correct rendering in the configuration I tested. The address boundary and the underlying cause are still unknown.
The standalone reproducer demonstrates one case. With an image backed by a high-address memory allocation, enabling VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_CAPTURE_REPLAY_BIT causes device loss. Disabling the flag lets the dispatch complete and restores correct readback in the reproducer. If the image is backed by a low-address memory allocation, the dispatch succeeds even with the flag enabled.
In the renderer, disabling the flag avoided the observed device loss, but the skybox flickered. Moving the skybox to a low-address allocation eliminated the flicker even with the flag enabled. I have not isolated which operation causes that remaining corruption at higher addresses.
Standalone reproducer
The program creates a 1280 × 720 VK_FORMAT_R32_UINT image and accesses it through a VK_DESCRIPTOR_TYPE_STORAGE_IMAGE descriptor in the resource heap. A compute dispatch queries the image dimensions and writes their product to each texel. The expected readback at texel (0,0) is 921600. The CMake option -DSHADER_DEBUG_INFO=ON includes shader debug information. Debug builds log address binding events through VK_EXT_device_address_binding_report.
Run the following cases with the same executable. Substitute the path to the built executable for Repro.exe. The size argument controls the backing allocation, while the image dimensions stay fixed. Capture/replay is enabled by default for that allocation.
.\Repro.exe --image-mem-alloc-mib 512
.\Repro.exe --image-mem-alloc-mib 512 --image-mem-no-capture-replay
.\Repro.exe --image-mem-alloc-mib 256
| Image backing allocation | Capture/replay flag | Result on the tested system |
|---|---|---|
| 512 MiB | On | The allocation’s base address landed above 4 GiB. vkDeviceWaitIdle() returned VK_ERROR_DEVICE_LOST. |
| 512 MiB | Off | The allocation’s base address landed above 4 GiB. The dispatch completed and readback returned 921600. |
| 256 MiB | On | The allocation’s base address landed below 4 GiB. The dispatch completed and readback returned 921600. |
vkQueueSubmit2() can return VK_SUCCESS even when vkDeviceWaitIdle() subsequently reports VK_ERROR_DEVICE_LOST. An earlier version of the reproducer ignored the wait result and reported a readback value of 0 after device loss. Checking the wait result confirmed that those runs also lost the device. Readback is evaluated only after a successful wait. Audio buzzed during failing runs. The allocation sizes above produced the stated address ranges on my setup. Other configurations may place allocations differently, so the reported base address is useful when comparing results.
Renderer observations
-
Allocation placement: Examples of working backing allocation bases include
0x07000000and0x17c00000. Examples associated with failures include0x260200000,0x2a0200000, and0x320200000. In one reduced renderer test, increasing the backing allocation from 128 MiB to 256 MiB triggered device loss with the original image binding offsets retained. In other controlled runs, shifting the affected image’s binding offset by 16 MiB also shifted the fault address reported by NVIDIA Aftermath after device loss by 16 MiB. The fault address remained outside the reported backing allocation range. -
Low-address configuration: Reducing the resource set and cubemap size let the renderer use a small image allocation at a lower address. In that reduced configuration, the skybox rendered correctly and Nsight Graphics was able to launch the renderer and capture frames, which it was not in configurations with a flickering skybox. Both the equirectangular-to-cube conversion of the skybox and the tone mapping pass use storage image writes. This configuration required removing resources and assets from the renderer.
-
Affected operations: I observed failures on both storage image reads and writes, in compute and fragment shaders, with 2D and 2D-array storage images. Both bindful and bindless heap mappings exhibited the issue.
-
Working paths: Sampled image accesses through the resource heap and storage buffer reads and writes worked in the tested cases. The same storage images worked with classic Vulkan descriptor sets. The corresponding D3D12 renderer logic also worked.
-
Capture/replay and buffer addresses: The renderer originally allocated memory with both
VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BITandVK_MEMORY_ALLOCATE_DEVICE_ADDRESS_CAPTURE_REPLAY_BIT. In my buffer tests, the device address bit alone yielded coincident reported base addresses for the buffer and its backing memory. Adding the capture/replay bit produced a separate buffer mapping in the0xe000000000region. How the flag affects image texel addressing remains unknown.
Environment
- GPU: NVIDIA GeForce RTX 2060 Mobile, 6 GiB VRAM
- OS: Windows 11
- Vulkan SDKs tested: 1.4.341, 1.4.357, 1.4.363
- NVIDIA drivers tested: 610.74, 616.92, 617.14, 621.01, 621.14
- Shaders: HLSL compiled to SPIR-V with DXC
- The issue reproduces with and without Vulkan Validation Layers enabled.
Validation and GPU fault diagnostics
The GPU descriptor dump resolves the shader binding to the intended storage image, and HLSL GetDimensions() returns the expected dimensions. NVIDIA Aftermath attributes the MMU translation fault to the storage image access. In the disassembly, the fault is associated with OpImageWrite or, in the renderer read test, OpImageRead. No fault was observed on the preceding OpLoad of the image object. No relevant validation error was emitted before the GPU fault in these runs.
Aftermath reports a failure to translate a GPU virtual address in runs that lose the device. Resource tracking through both the Monitor and SDK integration provided no resource association for the fault address. One example is the raw SSAO storage read at SSAO.comp.hlsl:53, which reported a read fault at 0x000000e301ec2000.
Reproducer descriptor dump
Vulkan Debug Message
[Severity] INFO
[Type] VALIDATION
[Message] vkCmdDispatch(): [Dump Descriptor] (VkCommandBuffer 0x1fa89c870d0, VkPipeline 0x1fa898610d0)
- vkCmdBindResourceHeapEXT last bound the resource heap to [0xe000000000, 0xe000017a20) (reserved range: 96768 bytes at offset 0)
- VkBuffer 0x1fa89ae54d0[Buffer: Descriptor Heap], size: 96800 bytes, range: [0xe000000000, 0xe000017a20)
- Last bound pipeline: VkPipeline 0x1fa898610d0 (VK_PIPELINE_BIND_POINT_COMPUTE)
- Shader descriptors:
[VK_SHADER_STAGE_COMPUTE_BIT] VkShaderModule 0x1fa897e2a10
- SPIR-V Set 0:
- SPIR-V Binding 0 ["gStorageImage"]
- specified in pMappings[0] - VK_DESCRIPTOR_MAPPING_SOURCE_HEAP_WITH_PUSH_INDEX_EXT
- pushOffset: 0, heapOffset: 0x0, heapIndexStride: 1, heapArrayStride: 32
- pushData[0:3] = 96768
- pushIndex offset: 96768 * 1
- Resource Heap address: 0xe000017a00
- Found descriptor mapped to: VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, VK_IMAGE_VIEW_TYPE_2D, VK_FORMAT_R32_UINT, VkImage 0x1fa89ad0cd0[Image: Test Image]
- Descriptor size: 32 (VK_DESCRIPTOR_TYPE_STORAGE_IMAGE)