Getting weird device lost only when fast linking graphics pipeline

[alex.rou.sg]

I am using VK_EXT_graphics_pipeline_library with dynamic rendering and I am getting a device lost after a vkQueuePresentKHR,
the only pipeline bound is drawing a textured circle, nothing too fancy,

aftermath reports:
MMU fault detected during a Read of address 0x00000000000000000000000068ec0000.
Access originating from Graphics Processing Cluster client failed with the following error:
Failed to translate the virtual address.
A MMU fault in the Graphics Processing Cluster may indicate texture fetch or other shader memory issues.

however the weird thing is that this only happens with shader cache disabled (NV control panel) or if the shader wasn’t cached, with cache enabled, the program runs normally on the second startup onwards

This is on windows 10 with a 1080ti
Tested Drivers: 537.96, 546.01

API Dump also reports correct binding of the descriptor set and image so it feels like a likely driver bug.

[alex.rou.sg]

so I’ve managed to find a way to cause this on demand, apparently fast linking pipelines is broken
if my graphics pipeline was created with VK_PIPELINE_CREATE_LINK_TIME_OPTIMIZATION_BIT_EXT it doesn’t crash.

[alex.rou.sg]

While debugging an unrelated issue, I found out I was running into VUID-VkGraphicsPipelineCreateInfo-pLibraries-06621
fixing that and I am unable to reproduce this on 546.33