I have got strange E_OUTOFMEMORY error when calling ID3D12Device::CreateGraphicsPipelineState trying to use constant buffers in VS using shader model 5.1. The error only happens with NVidia card (I am testing with 1070), but not other brands of GPU so it should be driver issue.
The following steps can be use to reproduce the issue:
- Get the MS d3d12 sample “D3D12HelloWorld” from https://github.com/Microsoft/DirectX-Graphics-Samples
- Open “D3D12HelloConstBuffers.cpp” in project “D3D12HelloConstBuffers”, and change the VS shader compile target (the line that calls “D3DCompileFromFile(…VSMain…)”) from “vs_5_0” to “vs_5_1”
- Run the project “D3D12HelloConstBuffers” using Debug configuration. The VSMain compilation is fine, but CreateGraphicsPipelineState throws E_OUTOFMEMORY error.
Also note that in this particular case, the error only happens when “D3DCOMPILE_SKIP_OPTIMIZATION” is specified (i.e. in Debug build). However in some cases of our own tests the error will be thrown regardless of the flag.