You’re not alone, I have this issue as well. It seems there might be some vsyncing issues with X11 (and XWayland) which seems to become worse with any Vulkan application running with vsync enabled see:
Other users in this thread also mentioned similar issues with X11:
Maybe some of the steps suggested by the moderators might help?