I have this strange problem with nvidia-340 driver (version 340.107-0ubuntu0.18.04.3) in Ubuntu 18.04. When I have playback window open on MythTV’s mythfrontend, and I use accelerated graphics profile (VDPAU or OpenGL), and I put another window on having some area of almost black RGB color (2,2,2), that color will be transparent showing image of the playback window behind (even if there is/are yet another window between them).
It happens mainly with MythTV’s mythfronted in my experience. (MythTV version 29.1 from Mythbuntu PPA) It does not usually happen with Kodi, VLC or mplayer, but once I could make (2,2,2) pixels of MythTV’s mythfrontend transparent when stared playing some video by another player before opening mythfrontend for playback.
I have the same problem with another computer that has the same OS and driver but different NVIDIA graphics hardware. The problem was also present with older version of Ubuntu, nvidia-340 driver and MythTV.
The green background in the first picture of that article corresponds to the window that should display the video. All pixels in this window are set to the colorkey. The person in the front corresponds to other windows that (partially) obscure the window displaying the video.
The overlay is used for performance reasons and to prevent tearing: it allows the video application to update its video surface asynchronously from the rest of the display. The GPU composites the video on top of the screen’s contents wherever it finds the color key value. The video application is supposed to choose a color key value that is unlikely to show up anywhere else on the screen.
If the color key behavior is causing a problem for you, you can force the driver to disable use of the overlay by setting VDPAU_NVIDIA_NO_OVERLAY=1 in the environment. See Appendix G: VDPAU Support in the README for more details.
I do not want to compromise performance and see tearing. I guess overlay is good with mythfrontend’s own UI components, because their colors can be controlled. However, if there is a desktop environment involved, any color can occur. In my use it would be better, that mythtv’s video would be blank, if a window from another application has focus. (My playback window is full screen.) It would not have to playback video at all, only audio. Would it be possible to have the overlay only when the video player has focus? Or when the playback window is not overlapping any window that is higher in the window stack?