I am having the same issue as this person and that person, both of whom use Nvidia cards as well: As soon as I target a game window using Xcomposite in OBS, there is massive tearing in the game on-screen. The tear is always in the upper fifth of the image.
There is no tearing in the resulting video file if I record one. There might however be strange graphical artifacts (flickering, dimming of light sources in the game), when encoding with NVENC. The tearing and these artifacts might be two unrelated issues.
The artifacts are visible in this video, particularly pronounced when the character picks up credits. The strange darkened squares and flickering around the yellow credits aren’t supposed to be there and weren’t there on-screen while playing.
I’ve tested this mostly with Crimsonland and JYDGE, particularly the latter showing the weird artifacts in the video. Now I’m expanding to other games (Momodora and Owlboy) and it seems to appear everywhere.
The following makes no difference: enabling or disabling compositing in the desktop (KDE Plasma). Enabling/disabling sync to vblank and/or flipping in Nvidia OpenGL settings. Enabling or disabling vsync in the game.
Recording with Xcomposite using an Intel HD 540 iGPU does not produce the problem, but it’s broken in a different way (wrong colors unless I use “swap red and blue” and flickering when recording in 60 fps).
Nvidia driver: 390.25
Kernel: 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2
Distro: Debian 9
nvidia-bug-report.log.gz (101 KB)