nVIDIA bug causing black screen in OBS Studio (Linux Mint 20.1 Ulyssa)

Having issue with 64-bit Linux Mint 20.1, Cinnamon and OBS 26.1.1. Some windows with XComposite are black (desktop, web browser Brave, Chromium, etc.), some visible (terminal, Thunderbird, System settings, file manager, etc.). At the same time desktop with XHSM is working fine.

Having nVidia Quadro M2000M (GM107GLM). Allow flipping is disabled - didn’t help. Also that alpha channel Mesa solution option under XComposite settings didn’t help.
nvidia-xconfig --composite didn’t help either :(

$ inxi -xxxG
Graphics:
Device-1: NVIDIA GM107GLM [Quadro M2000M] vendor: Lenovo driver: nvidia v: 460.39
bus ID: 01:00.0 chip ID: 10de:13b0
Display: x11 server: X.Org 1.20.9 driver: nvidia
unloaded: fbdev,modesetting,nouveau,vesa resolution: 2560x1440~60Hz
OpenGL: renderer: Quadro M2000M/PCIe/SSE2 v: 4.6.0 NVIDIA 460.39 direct render: Yes

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39 Driver Version: 460.39 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro M2000M Off | 00000000:01:00.0 On | N/A |
| N/A 44C P8 N/A / N/A | 394MiB / 4035MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 18890 G /usr/lib/xorg/Xorg 186MiB |
| 0 N/A N/A 19251 G cinnamon 38MiB |
| 0 N/A N/A 19655 G ...gAAAAAAAAA --shared-files 12MiB |
| 0 N/A N/A 19806 G ...AAAAAAAAA= --shared-files 14MiB |
| 0 N/A N/A 19910 G ...96AB038DB6D131158B746C3CC 3MiB |
| 0 N/A N/A 20058 G ...token=1743612660864995289 12MiB |
| 0 N/A N/A 26848 G ...AAAAAAAAA= --shared-files 114MiB |
+-----------------------------------------------------------------------------+

Another similar issue:

This issue is outlined in detail over in GitHub in this issue: Black preview screen in Linux Mint · Issue #4259 · obsproject/obs-studio · GitHub

It seems like many users are experiencing this issue, But the TL;DR:

Having a nVIDIA Ge Force 1080Ti with the latest drivers (460) and when I load OBS (v26.1.1) a black screen appears with no preview available (screenshot linked below).

None of the steps mentioned in the issue work. Is there a way to fix this? If not, is this a known bug that’s being worked on? Seems like it was a known issue here but the fix doesn’t seem to have helped all cards: NVIDIA DRIVERS Linux Display Driver - x86

Screenshot: https://obsproject.com/forum/attachments/screenshot-from-2021-02-17-21-31-51-png.67703/

Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.

Here it is:
nvidia-bug-report.log.gz (495.8 KB)

Actually it would be useful to suggest to run:
LC_ALL=C nvidia-bug-report.sh
… as many users are using Linux in their native language, which will affect also that script (dates, etc.).

E.g. compare in Linux terminal following commands:
date
with
LC_ALL=C date
This has difference, when running user environment in any other language, than English.

Hi, I have also the same issue for a long time.
So I also upload the bug report from my side.
I hope it will be an additional help to solve.

I use CentOS8 with GeForce GTX 1650.

nvidia-bug-report.log.gz (449.3 KB)

Thanks for proving the logs, though I suspected some issue with prime, which isn’t the case. To rule out a simple issue, please set

__GL_MaxFramesAllowed=1

in your system environment, reboot and check if that resolves the black windows?

I set the environment variable in /etc/profile.d/nvidia.sh to export the value.
Then I restarted my system. But still OBS Studio cannot capture the window.
II also took the log of OBS Studio and I hope it can be some help.

nvidia-bug-report.log.gz (752.9 KB)
-------------------------------- Output of OBS-Studio --------------------
info: Switched to scene ‘Scene’
info: ------------------------------------------------
info: Loaded scenes:
info: [adv-ss] started
info: - scene ‘Scene’:
info: - source: ‘Window Capture (Xcomposite)’ (xcomposite_input)
info: ------------------------------------------------
info: [obs-websocket] WSServer::start: Not locked to IPv4 bindings
info: [obs-websocket] server started successfully on port 4444
info: [obs-websocket] io thread started
Attempted path: share/obs/obs-studio/images/overflow.png
Attempted path: /usr/share/obs/obs-studio/images/overflow.png
info: adding 42 milliseconds of audio buffering, total audio buffering is now 42 milliseconds (source: Mic/Aux)

info: xcompcap: [window-capture: ‘Window Capture (Xcomposite)’] update settings:
title: OBS 26.1.2-35-gbb6b1e53f-modified (linux) - Profile: Untitled - Scenes: Untitled
class: obs
Bit depth: 24
Found proper GLXFBConfig (in 9): yes

error: xcompcap: cleanup glXReleaseTexImageEXT failed: GLXBadPixmap
error: xcompcap: glXCreatePixmap failed: BadAlloc (insufficient resources for operation)

Hi,
I tried to change my video card to my old GTX-560 Ti and rolled back the driver to ver.390.
Unlike GTX1650 with driver 460, it can capture the Chrome screen.
But with the card, obs consumes unacceptable CPU power.

nvidia-bug-report.log.gz (479.0 KB)

Following log is the gdb result with GTX1650 when the bug occured.

obs.log.gz (9.5 KB)

What I’ve tested so far:

  • nvidia-driver-460 and nvidia-driver-390 gave same result (not working): the export __GL_MaxFramesAllowed=1 variable via /etc/profile.d/nvidia.sh didn’t help, also nvidia-xconfig --composite as superuser didn’t help. Tried with and without. Allow flipping is also disabled. HDMI audio worked, and 2560x1440 full screen video in TV worked fine (CPU load 15…25%). Having also official Ubuntu graphics driver repository added.
  • nouveau (open-source nVidia driver by community) - here OBS Studio worked fine (xcomposite windows were captured) but CPU load with 2560x1440 full screen video in TV took 100% CPU and cannot use. Although I used newest stable Mesa repository (ppa:kisak/kisak-mesa).

Here I attach also these bug reports I’ve created.
nvidia-driver-390:

Finally installed nvidia-driver-460 back and so it remained for now.

Looks like for me Chromium, Brave windows are black. Firefox is captured, other apps windows (as many as I tested) are also usually captured. Still found Hypnotix app (free TV-channels) were not captured initially, but when I resized it a bit - then it was captured. However, in case of Chromium, etc. that window resizing trick didn’t work. I installed nvidia-driver-460 back.

Have to admit, that at the same time Simple Screen Recorder will capture everything just fine. But OBS Studio cannot.

Also was interesting, that when I started Chromium or Brave in application mode from terminal (chromium --app=URL), then OBS could capture the window! But I am using also the app named WebCatalog, which will create sandboxed apps by URL and these windows were still black. In another words, WebCatalog can make any URL as desktop app. WebCatalog is also using Chromium.

… and also nvidia-460 logs

id like to report a similar issue awhile back when upgrading to a newer Mesa driver (1.20.x)

at first my work around was the use Screen Capture, i finally solved the issue by discovering it was the Mesa version and reverted to v1.19.x

i believe its MESA combined with Prime/Nvidia that has this issue.
other workarounds or solving this, i have no idea. atleast if you are able to use a different MESA driver version you can test this further.

here is a video example captured with OBS Window Capture:

and some specs:

uname -a
Linux Origin-EON-17S 4.15.0-123-generic #126-Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
LSB Version: core-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: LinuxMint
Description: Linux Mint 19.3 Tricia
Release: 19.3
Codename: tricia

nvidia-smi
Thu Nov 12 05:51:56 2020
±----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1070 Off | 00000000:01:00.0 Off | N/A |
| N/A 53C P0 37W / N/A | 90MiB / 8119MiB | 2% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

vulkaninfo

VULKANINFO

Vulkan Instance Version: 1.2.154

inxi -Fxz
System: Host: Origin-EON-17S Kernel: 4.15.0-123-generic x86_64 bits: 64 compiler: gcc
v: 7.5.0 Desktop: MATE 1.24.1 Distro: Linux Mint 19.3 Tricia
base: Ubuntu 18.04 bionic
CPU: Topology: Quad Core model: Intel Core i7-3940XM bits: 64 type: MT MCP
arch: Ivy Bridge rev: 9 L2 cache: 8192 KiB
flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 51088
Speed: 1807 MHz min/max: 1200/3900 MHz Core speeds (MHz): 1: 1198 2: 1197 3: 1197
4: 1197 5: 1198 6: 1197 7: 1197 8: 1197
Graphics: Device-1: Intel 3rd Gen Core processor Graphics vendor: CLEVO/KAPOK driver: i915
v: kernel bus ID: 00:02.0
Device-2: NVIDIA GP104M [GeForce GTX 1070 Mobile] vendor: CLEVO/KAPOK
driver: nvidia v: 450.80.02 bus ID: 01:00.0
Display: x11 server: X.Org 1.20.8 driver: modesetting,nvidia
resolution: 1920x1080~60Hz, 1920x1080~60Hz
OpenGL: renderer: GeForce GTX 1070/PCIe/SSE2 v: 4.6.0 NVIDIA 450.80.02
direct render: Yes

this is my CURRENT
inxi -xxxG
Graphics: Device-1: Intel 3rd Gen Core processor Graphics vendor: CLEVO/KAPOK driver: i915
v: kernel bus ID: 00:02.0 chip ID: 8086:0166
Device-2: NVIDIA GP104M [GeForce GTX 1070 Mobile] vendor: CLEVO/KAPOK
driver: nvidia v: 450.119.03 bus ID: 01:00.0 chip ID: 10de:1be1
Display: x11 server: X.Org 1.19.6 driver: modesetting,nvidia
alternate: fbdev,nouveau,vesa compositor: marco v: 1.24.1
resolution: 1920x1080~60Hz, 1920x1080~60Hz
OpenGL: renderer: GeForce GTX 1070/PCIe/SSE2 v: 4.6.0 NVIDIA 450.119.03
direct render: Yes

in comparison to GL, Gaming was impossible to notice
however Benchmarks showed major performace reduction

glmark2 -s 1920x1080
Mesa 1.19.x

                              glmark2 Score: 7719 

=======================================================
Mesa 1.20.x

                              glmark2 Score: 208 

=======================================================

how this relates to the original topic??
Black screen window capture in OBS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^