glXSwapBuffers() gobbling up a full CPU core when vsync is off

I did some experiments with vsync:

VSYNC ON: 60 fps, 6% of 1 CPU core.

VSYNC OFF: 150 fps, 100% of 1 CPU core.

I profiled the run, and with vsync off, I see glXSwapBuffers() take 96% of the CPU cycles.

The FPS delta (150/60) is just a factor 2.5 so it would be doing, at most, 2.5 times the work on the CPU. But the CPU usage goes from nothing to 100%.

What is going on here?

OS: Ubuntu
OpenGL: 3.2 Core Profile
Driver: 4.6.0 NVIDIA 455.23.05

Why does the buffer swap go from no-cost to a 100% CPU cost?

UPDATE

Adding a glFinish() before the swap buffers does not help.

Using __GL_YIELD=USLEEP will however, make CPU usage drop from 100% of a core, to 20% of a core.