NVIDIA DRM/KMS: Gamescope embedded mode fails atomic flip on headless RTX 4090 VM despite valid EDID and working Weston KMS output

Graphics / Linux Linux
, , ,
1

I am trying to build a headless cloud-gaming setup using NVIDIA DRM/KMS, Sunshine/Moonlight, and a virtual EDID. The goal is to eventually support 3840x2160 at 120 Hz with HDR.

The machine is an Ubuntu 24.04 VM with an NVIDIA RTX 4090. I am using nvidia-drm.modeset=1 and a custom EDID injected via kernel cmdline. The virtual connector appears as DP-1, and the EDID exposes 4K120 plus HDR static metadata.

Kernel cmdline includes:

drm.edid_firmware=DP-1:edid/virtual-4k120-hdr.bin video=DP-1:e nvidia-drm.modeset=1 nvidia-drm.fbdev=1

The connector is detected correctly:

DP-1 connected
modes include 3840x2160, 1920x1080
EDID includes:
- VIC 118: 3840x2160 120 Hz
- VIC 97: 3840x2160 60 Hz
- BT2020RGB
- SMPTE ST2084 / PQ
- HLG
- HDR Static Metadata Type 1

Weston can drive the output through the DRM backend, so the connector/EDID path is not totally broken. However, Gamescope embedded mode fails its atomic flip/commit path.

I tested two Gamescope builds:

  1. Older package build around 3.14.x

  2. Nix/NixGL Gamescope 3.16.23 with NVIDIA 580.126.09 userspace libraries

The newer Gamescope successfully initializes Vulkan, sees the RTX 4090, opens /dev/dri/card1, selects DP-1, reads the HDR EDID/colorimetry, and selects the desired mode. Example log:

gamescope version 3.16.23
vulkan: selecting physical device 'NVIDIA GeForce RTX 4090'
vulkan: physical device supports DRM format modifiers
drm: opening DRM node '/dev/dri/card1'
drm: Connector DP-1 -> SAM - Virtual4KHDR
drm: selecting connector DP-1
drm: selecting mode 3840x2160@120Hz

But it repeatedly fails with:

xwm: Failed to prepare 1-layer flip (Invalid argument), trying again with previous mode if modeset needed
xwm: Failed to prepare 1-layer flip entirely: Invalid argument

This happens at 4K120 and also at 4K60. It happens with an Xwayland client (xterm) and also with a simple Wayland client (weston-simple-shm). So this does not appear to be specifically an Xwayland issue. In both cases, the real DRM state remains at the existing 3840x2160 60 Hz mode rather than being successfully owned by Gamescope. The 4K60 test still repeatedly selected 3840x2160@60Hz and failed the same 1-layer flip with Invalid argument.

I also fixed an initial NixGL/GBM path issue by creating /run/opengl-driver/lib -> /nix/store/...-nvidia-x11-580.126.09-nixGL/lib, and confirmed that the missing GBM loader error disappeared. But the atomic flip failure remains. The system has both Nix and host NVIDIA GBM libraries available, including nvidia-drm_gbm.so.

Separately, I wrote small libdrm atomic test programs to probe the NVIDIA private color/HDR properties.

The following properties are exposed:

CRTC:
- NV_CRTC_REGAMMA_TF

Plane:
- COLOR_ENCODING
- NV_INPUT_COLORSPACE
- NV_HDR_STATIC_METADATA
- NV_PLANE_DEGAMMA_TF

Connector:
- Colorspace
- HDR_OUTPUT_METADATA

Testing individual properties showed:

PASS:
- CRTC NV_CRTC_REGAMMA_TF = PQ
- plane COLOR_ENCODING = BT.2020
- plane NV_INPUT_COLORSPACE = scRGB
- plane NV_INPUT_COLORSPACE = BT.2100 PQ
- plane NV_PLANE_DEGAMMA_TF = PQ

FAIL:
- connector Colorspace = BT2020_RGB
- connector HDR_OUTPUT_METADATA blob
- plane NV_HDR_STATIC_METADATA blob

A metadata-free HDR-ish atomic commit also succeeded when setting:

CRTC NV_CRTC_REGAMMA_TF = PQ
Plane COLOR_ENCODING = BT.2020
Plane NV_INPUT_COLORSPACE = BT.2100 PQ
Plane NV_PLANE_DEGAMMA_TF = PQ

The key breakthrough was that NV_INPUT_COLORSPACE did not appear to stick with the existing framebuffer, but it did stick when I committed my own 10-bit XB30 framebuffer. With the custom 10-bit framebuffer active, I observed:

NV_CRTC_REGAMMA_TF = 2
COLOR_ENCODING = 2
NV_INPUT_COLORSPACE = 2
NV_PLANE_DEGAMMA_TF = 2

So it looks like the NVIDIA driver can accept the private PQ/BT.2020 path when the framebuffer format is compatible.

However, Gamescope embedded mode appears to be producing an atomic state that NVIDIA rejects with EINVAL, even before HDR is enabled.

My questions:

  1. Is there a known limitation or required atomic property combination for Gamescope-style embedded KMS compositors on NVIDIA when using virtual/headless DP connectors?

  2. Are there NVIDIA-specific requirements for framebuffer format/modifiers when performing a direct scanout/flip at 3840x2160, especially with Gamescope?

  3. Is the Failed to prepare 1-layer flip entirely: Invalid argument likely caused by an unsupported modifier, plane format, CRTC/plane property combination, or something else?

  4. For HDR output on NVIDIA DRM, should userspace prefer the standard connector properties (HDR_OUTPUT_METADATA, Colorspace) or NVIDIA private plane/CRTC properties (NV_INPUT_COLORSPACE, NV_PLANE_DEGAMMA_TF, NV_CRTC_REGAMMA_TF, etc.)?

  5. Is there a recommended reference implementation or sample userspace code for setting NVIDIA HDR/PQ/BT.2020 output through DRM/KMS?

I can provide full modetest -p, EDID, Gamescope logs, and my small libdrm probes if helpful.

Thanks.