BLAS building fails with a large buffer without an irrelevant flag

For some reason building a BLAS with a large buffer (exact size seems to vary but consistently above 2MB) causes a device lost in a later call, this is somehow fixed when creating the buffer and allocating memory with

VK_BUFFER_CREATE_DEVICE_ADDRESS_CAPTURE_REPLAY_BIT

I’m new to Vulkan, but I can’t seem to find a reason for this flag to be set. I first found this behavior when trying to debug issue with Nsight (enabling Device Address C++ Support made this go away), Nsight appears to be forcing this flag on.

Does anyone know what might be causing this?