After debugging a Wine bug involving a hang in glXCreateContext() I have a guess that the root cause lies in a lock ordering problem in the NVidia OpenGL driver.
We recently started locking the X11 display [with XLockDisplay()] while calling glXCreateContext(), namely so that we can correctly catch X errors from that function, and this change seems to have introduced a deadlock in a certain game. I have observed that when the deadlock happens, another thread—which previously had an active GLX context, but deleted it—is sleeping after having called pthread_exit().
My guess therefore (which is pure speculation) is that:
- the NVidia GLX driver is submitting a pthread cleanup via pthread_cleanup_push(),
- it is taking an internal lock in both that and glXCreateContext(),
- inside of glXCreateContext() it calls XLockDisplay(),
which results in lock inversion when glXCreateContext() is called with the display already locked.
Is this explanation plausible, or something similar? If not I can try to provide reproduction instructions (or a backtrace).
More information (most of which is rather specific to Wine) is available here: