nVIDIA bug causing black screen in OBS Studio (Linux Mint 20.1 Ulyssa)

zeroconf · March 24, 2021, 11:31pm

Having issue with 64-bit Linux Mint 20.1, Cinnamon and OBS 26.1.1. Some windows with XComposite are black (desktop, web browser Brave, Chromium, etc.), some visible (terminal, Thunderbird, System settings, file manager, etc.). At the same time desktop with XHSM is working fine.

Having nVidia Quadro M2000M (GM107GLM). Allow flipping is disabled - didn’t help. Also that alpha channel Mesa solution option under XComposite settings didn’t help.
nvidia-xconfig --composite didn’t help either :(

$ inxi -xxxG
Graphics:
Device-1: NVIDIA GM107GLM [Quadro M2000M] vendor: Lenovo driver: nvidia v: 460.39
bus ID: 01:00.0 chip ID: 10de:13b0
Display: x11 server: X.Org 1.20.9 driver: nvidia
unloaded: fbdev,modesetting,nouveau,vesa resolution: 2560x1440~60Hz
OpenGL: renderer: Quadro M2000M/PCIe/SSE2 v: 4.6.0 NVIDIA 460.39 direct render: Yes

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39 Driver Version: 460.39 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro M2000M Off | 00000000:01:00.0 On | N/A |
| N/A 44C P8 N/A / N/A | 394MiB / 4035MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 18890 G /usr/lib/xorg/Xorg 186MiB |
| 0 N/A N/A 19251 G cinnamon 38MiB |
| 0 N/A N/A 19655 G ...gAAAAAAAAA --shared-files 12MiB |
| 0 N/A N/A 19806 G ...AAAAAAAAA= --shared-files 14MiB |
| 0 N/A N/A 19910 G ...96AB038DB6D131158B746C3CC 3MiB |
| 0 N/A N/A 20058 G ...token=1743612660864995289 12MiB |
| 0 N/A N/A 26848 G ...AAAAAAAAA= --shared-files 114MiB |
+-----------------------------------------------------------------------------+

Another similar issue:

This issue is outlined in detail over in GitHub in this issue: Black preview screen in Linux Mint · Issue #4259 · obsproject/obs-studio · GitHub

It seems like many users are experiencing this issue, But the TL;DR:

Having a nVIDIA Ge Force 1080Ti with the latest drivers (460) and when I load OBS (v26.1.1) a black screen appears with no preview available (screenshot linked below).

None of the steps mentioned in the issue work. Is there a way to fix this? If not, is this a known bug that’s being worked on? Seems like it was a known issue here but the fix doesn’t seem to have helped all cards: Linux Display Driver - x86 | 340.108 | Linux 32-bit | NVIDIA

Screenshot: https://obsproject.com/forum/attachments/screenshot-from-2021-02-17-21-31-51-png.67703/

generix · March 25, 2021, 10:02am

Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.

zeroconf · March 26, 2021, 1:49am

Here it is:
nvidia-bug-report.log.gz (495.8 KB)

Actually it would be useful to suggest to run:
LC_ALL=C nvidia-bug-report.sh
… as many users are using Linux in their native language, which will affect also that script (dates, etc.).

E.g. compare in Linux terminal following commands:
date
with
LC_ALL=C date
This has difference, when running user environment in any other language, than English.

nshimizu · March 26, 2021, 2:36am

Hi, I have also the same issue for a long time.
So I also upload the bug report from my side.
I hope it will be an additional help to solve.

I use CentOS8 with GeForce GTX 1650.

nvidia-bug-report.log.gz (449.3 KB)

generix · March 26, 2021, 11:40am

Thanks for proving the logs, though I suspected some issue with prime, which isn’t the case. To rule out a simple issue, please set

__GL_MaxFramesAllowed=1

in your system environment, reboot and check if that resolves the black windows?

nshimizu · March 26, 2021, 1:38pm

I set the environment variable in /etc/profile.d/nvidia.sh to export the value.
Then I restarted my system. But still OBS Studio cannot capture the window.
II also took the log of OBS Studio and I hope it can be some help.

nvidia-bug-report.log.gz (752.9 KB)
-------------------------------- Output of OBS-Studio --------------------
info: Switched to scene ‘Scene’
info: ------------------------------------------------
info: Loaded scenes:
info: [adv-ss] started
info: - scene ‘Scene’:
info: - source: ‘Window Capture (Xcomposite)’ (xcomposite_input)
info: ------------------------------------------------
info: [obs-websocket] WSServer::start: Not locked to IPv4 bindings
info: [obs-websocket] server started successfully on port 4444
info: [obs-websocket] io thread started
Attempted path: share/obs/obs-studio/images/overflow.png
Attempted path: /usr/share/obs/obs-studio/images/overflow.png
info: adding 42 milliseconds of audio buffering, total audio buffering is now 42 milliseconds (source: Mic/Aux)

info: xcompcap: [window-capture: ‘Window Capture (Xcomposite)’] update settings:
title: OBS 26.1.2-35-gbb6b1e53f-modified (linux) - Profile: Untitled - Scenes: Untitled
class: obs
Bit depth: 24
Found proper GLXFBConfig (in 9): yes

error: xcompcap: cleanup glXReleaseTexImageEXT failed: GLXBadPixmap
error: xcompcap: glXCreatePixmap failed: BadAlloc (insufficient resources for operation)

nshimizu · March 28, 2021, 1:02am

Hi,
I tried to change my video card to my old GTX-560 Ti and rolled back the driver to ver.390.
Unlike GTX1650 with driver 460, it can capture the Chrome screen.
But with the card, obs consumes unacceptable CPU power.

nvidia-bug-report.log.gz (479.0 KB)

Following log is the gdb result with GTX1650 when the bug occured.

obs.log.gz (9.5 KB)

zeroconf · March 31, 2021, 11:26pm

What I’ve tested so far:

nvidia-driver-460 and nvidia-driver-390 gave same result (not working): the export __GL_MaxFramesAllowed=1 variable via /etc/profile.d/nvidia.sh didn’t help, also nvidia-xconfig --composite as superuser didn’t help. Tried with and without. Allow flipping is also disabled. HDMI audio worked, and 2560x1440 full screen video in TV worked fine (CPU load 15…25%). Having also official Ubuntu graphics driver repository added.
nouveau (open-source nVidia driver by community) - here OBS Studio worked fine (xcomposite windows were captured) but CPU load with 2560x1440 full screen video in TV took 100% CPU and cannot use. Although I used newest stable Mesa repository (ppa:kisak/kisak-mesa).

Here I attach also these bug reports I’ve created.
nvidia-driver-390:

nvidia-390-bug-report.log.gz (144.4 KB)
obs-nvidia-driver-390.gz (2.4 KB)

Finally installed nvidia-driver-460 back and so it remained for now.

Looks like for me Chromium, Brave windows are black. Firefox is captured, other apps windows (as many as I tested) are also usually captured. Still found Hypnotix app (free TV-channels) were not captured initially, but when I resized it a bit - then it was captured. However, in case of Chromium, etc. that window resizing trick didn’t work. I installed nvidia-driver-460 back.

Have to admit, that at the same time Simple Screen Recorder will capture everything just fine. But OBS Studio cannot.

Also was interesting, that when I started Chromium or Brave in application mode from terminal (chromium --app=URL), then OBS could capture the window! But I am using also the app named WebCatalog, which will create sandboxed apps by URL and these windows were still black. In another words, WebCatalog can make any URL as desktop app. WebCatalog is also using Chromium.

zeroconf · April 1, 2021, 12:02am

… and also nvidia-460 logs

nvidia-bug-report.log.gz (495.8 KB)
obs-nvidia-driver-460.gz (2.4 KB)

arrowgent · May 9, 2021, 8:42pm

id like to report a similar issue awhile back when upgrading to a newer Mesa driver (1.20.x)

at first my work around was the use Screen Capture, i finally solved the issue by discovering it was the Mesa version and reverted to v1.19.x

i believe its MESA combined with Prime/Nvidia that has this issue.
other workarounds or solving this, i have no idea. atleast if you are able to use a different MESA driver version you can test this further.

here is a video example captured with OBS Window Capture:

External Media

and some specs:

uname -a
Linux Origin-EON-17S 4.15.0-123-generic #126-Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
LSB Version: core-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: LinuxMint
Description: Linux Mint 19.3 Tricia
Release: 19.3
Codename: tricia

vulkaninfo

VULKANINFO

Vulkan Instance Version: 1.2.154

inxi -Fxz
System: Host: Origin-EON-17S Kernel: 4.15.0-123-generic x86_64 bits: 64 compiler: gcc
v: 7.5.0 Desktop: MATE 1.24.1 Distro: Linux Mint 19.3 Tricia
base: Ubuntu 18.04 bionic
CPU: Topology: Quad Core model: Intel Core i7-3940XM bits: 64 type: MT MCP
arch: Ivy Bridge rev: 9 L2 cache: 8192 KiB
flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 51088
Speed: 1807 MHz min/max: 1200/3900 MHz Core speeds (MHz): 1: 1198 2: 1197 3: 1197
4: 1197 5: 1198 6: 1197 7: 1197 8: 1197
Graphics: Device-1: Intel 3rd Gen Core processor Graphics vendor: CLEVO/KAPOK driver: i915
v: kernel bus ID: 00:02.0
Device-2: NVIDIA GP104M [GeForce GTX 1070 Mobile] vendor: CLEVO/KAPOK
driver: nvidia v: 450.80.02 bus ID: 01:00.0
Display: x11 server: X.Org 1.20.8 driver: modesetting,nvidia
resolution: 1920x1080~60Hz, 1920x1080~60Hz
OpenGL: renderer: GeForce GTX 1070/PCIe/SSE2 v: 4.6.0 NVIDIA 450.80.02
direct render: Yes

this is my CURRENT
inxi -xxxG
Graphics: Device-1: Intel 3rd Gen Core processor Graphics vendor: CLEVO/KAPOK driver: i915
v: kernel bus ID: 00:02.0 chip ID: 8086:0166
Device-2: NVIDIA GP104M [GeForce GTX 1070 Mobile] vendor: CLEVO/KAPOK
driver: nvidia v: 450.119.03 bus ID: 01:00.0 chip ID: 10de:1be1
Display: x11 server: X.Org 1.19.6 driver: modesetting,nvidia
alternate: fbdev,nouveau,vesa compositor: marco v: 1.24.1
resolution: 1920x1080~60Hz, 1920x1080~60Hz
OpenGL: renderer: GeForce GTX 1070/PCIe/SSE2 v: 4.6.0 NVIDIA 450.119.03
direct render: Yes

in comparison to GL, Gaming was impossible to notice
however Benchmarks showed major performace reduction

glmark2 -s 1920x1080
Mesa 1.19.x

                              glmark2 Score: 7719

=======================================================
Mesa 1.20.x

                              glmark2 Score: 208

=======================================================

how this relates to the original topic??
Black screen window capture in OBS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

arrowgent · August 10, 2021, 10:00pm

i wanted to report some testing and findings here

previously working fine: Mesa 20.0.8, nvidia 460.x, xorg-xserver 1.19.6-1
update again to latest: Mesa 21.1.6, (nvidia 460.x), xorg-xserver 1.20.9-2

again same issue
however i went through testing…
i tested nvidia-drm.modeset=0 & nvidia-drm.modeset=1
not the issue
you can check modeset with sudo cat /sys/module/nvidia_drm/parameters/modeset

i checked my ~/.profile (/etc/envoirnment if global)

because i have nvidia-prime & an optimus laptop i also have set

export __NV_PRIME_RENDER_OFFLOAD=1
export __NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-0
export __GLX_VENDOR_LIBRARY_NAME=nvidia
export __VK_LAYER_NV_optimus=NVIDIA_only

what i found was using export __NV_PRIME_RENDER_OFFLOAD=1 is causing this issue

disabling this and rebooting we can test this

__NV_PRIME_RENDER_OFFLOAD=1 glxgears
then
__NV_PRIME_RENDER_OFFLOAD=1 obs
capture the glxgears window
PROBLEM FOUND.

and again setting __NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-0 glxgears
SAME PROBLEM

arrowgent · August 13, 2021, 3:01pm

NV_PRIME_RENDER_OFFLOAD was released in nvidia driver version 435 & 440
[Chapter 34. PRIME Render Offload]
[Chapter 35. PRIME Render Offload]

however
xorg does not have feature support for this yet!
According to this Phoronix article, the support will come with X.Org 21.1
[PRIME GPU Offloading Improvement For GLXVND Merged For X.Org Server 1.21 - Phoronix]

guess what is soon to release but is not released yet?
[X.Org Server 21.1 Will Aim To Release In The Coming Weeks - Phoronix]

testing using Mesa 21.1.6, (nvidia 460.x), xorg-xserver 1.20.9-2

$ __NV_PRIME_RENDER_OFFLOAD=1 glxgears
Running synchronized to the vertical refresh. The framerate should be
approximately the same as the monitor refresh rate.
10619 frames in 5.0 seconds = 2123.723 FPS
$ glxgears
Running synchronized to the vertical refresh. The framerate should be
approximately the same as the monitor refresh rate.
91118 frames in 5.0 seconds = 18223.490 FPS

zeroconf · October 25, 2021, 10:43pm

I have to admit, that I abandoned Linux Mint. Now using EndeavourOS with KDE Plasma 5 desktop environment and these issues are gone. Also other issues are gone, e.g. Cinnamon issues… There are plenty of Arch Linux based distros with rolling release model, that have friendly installer and later management, e.g. Manjaro Linux is more beginner friendly. Also EndeavourOS wiki has special section for nVidia. Basically also main nVidia installer article (EndeavourOS Nvidia Installer) is sufficient in most cases. Actually haven’t done any additional configuration - things are just working out-of-the-box.

Haven’t find any application window, that OBS cannot show - all works. I have many of apps installed. I’ve installed following packages:
yay -Syyuu obs-studio-tytan652 v4l2loopback-dkms v4l2loopback-utils luajit ffmpeg gphoto2 gphotofs --needed --noconfirm && yay -Scc --noconfirm
yay is used for package installation, similarly as apt in Linux Mint. v4l2loopback is needed for virtual cam, gPhoto is needed to turn DSLR camera into webcam, using ffmpeg. luajit is for Lua scripts used in OBS. tytan652 is the user, who regularly updates OBS Studio packages. Although compiled from source and it takes some time, the result is great. The obs-studio package is often outdated - updated much later. tytan652 has also some additions in OBS, more can be read at the package description itself in AUR.

Some information about used software, hardware.

Forum restrictions didn’t allow me post more links than 3 per post, sorry.

nVIDIA bug causing black screen in OBS Studio (Linux Mint 20.1 Ulyssa)

vulkaninfo

VULKANINFO

glmark2 -s 1920x1080 Mesa 1.19.x

======================================================= Mesa 1.20.x

glmark2 -s 1920x1080
Mesa 1.19.x

=======================================================
Mesa 1.20.x