Deadlock possibly caused by X display lock inversion in glXCreateContext()

After debugging a Wine bug involving a hang in glXCreateContext() I have a guess that the root cause lies in a lock ordering problem in the NVidia OpenGL driver.

We recently started locking the X11 display [with XLockDisplay()] while calling glXCreateContext(), namely so that we can correctly catch X errors from that function, and this change seems to have introduced a deadlock in a certain game. I have observed that when the deadlock happens, another thread—which previously had an active GLX context, but deleted it—is sleeping after having called pthread_exit().

My guess therefore (which is pure speculation) is that:

  • the NVidia GLX driver is submitting a pthread cleanup via pthread_cleanup_push(),
  • it is taking an internal lock in both that and glXCreateContext(),
  • inside of glXCreateContext() it calls XLockDisplay(),

which results in lock inversion when glXCreateContext() is called with the display already locked.

Is this explanation plausible, or something similar? If not I can try to provide reproduction instructions (or a backtrace).

More information (most of which is rather specific to Wine) is available here:

https://bugs.winehq.org/show_bug.cgi?id=53372#c9

I talked to zf about this off-forum and this is being tracked in bug 3777265.