If you create a vkDebugUtilsMessengerEXT and the pfnUserCallback is null, the validation layers do warn about this. However, it looks like the driver does not perform a null check on the function pointer and blindly follows the pointer, causing an X server crash. Basic example
While this is invalid, it should not crash the X server. The 460 drivers on Windows do not misbehave in any way with this situation, so this is specific to Linux only.
Ah yes I have that report and forgot to attach it. However, after a bit more investigation I don’t think it’s an invalid debug messenger callback causing the crash but an invalid swap chain creation. I’ll figure out exactly what conditions cause it and post a full reproduction program.
Alright, I have a program which will reliably crash the X server. I was mistaken, it wasn’t a null pointer to a debug messenger callback function, that was a red herring. It happens when creating a swapchain with the surfaceFormat being invalid. Vulkan validation layers do warn about the surface format is invalid. The full source code with comment at the place which causes the crash
This uses xcb, Vulkan 1.2.162, nVidia X server 450 and drivers 460.
The same program using Win32 on Windows will crash with a divide by zero error within nvogl.dll, which still isn’t perfect behavior but doesn’t completely kill the window manager. Probably safer on Windows because drivers are user space on Windows.