PRIME and PRIME Synchronization

It’s unclear to me what you mean when you refer to “dGPU’s buffers” and “iGPU’s buffers.” All of the buffers in this swap chain originate from the dGPU driver, although the buffers that the iGPU flips between for PRIME Synchronization are allocated in system memory in a format that the iGPU can understand.

There is an intermediate composition from the X screen’s primary surface into an intermediate video memory buffer (similarly to ForceFullCompositionPipeline = On) before asynchronously copying from that into the requested system memory back buffer, but that won’t add any additional latency because it all completes before the iGPU’s next vblank. This is done for performance reasons, as the composition step is done in the 3D channel, and we don’t want 3D applications to be blocked behind a relatively slow copy into system memory as they would if we composited directly into the system memory back buffer. Fermi GPUs lack this asynchronous copy support, so they composite directly into system memory at the expense of 3D performance for lack of a better option.

One issue may arise from the fact that OpenGL swaps after each composition, potentially adding more latency into the swap chain.

If you want to minimize input lag as much as possible today, your best bet is to set __GL_SYNC_TO_VBLANK=0. Naturally, the application won’t sync to vblank, but due to an implementation detail in PRIME, will not tear. Composition is essentially atomic, and incomplete frames will drop rather than tear. Under PRIME, GL Sync to VBlank has more to do with throttling the application to vblank while maintaining smoothness than it does with avoiding tearing.

Disabling GL Sync to VBlank should eliminate the potential additional input lag from the GL->PRIME->iGPU swapchain while maintaining tear-free, at the expense of power at framerates much higher than the refresh rate, or at the expense of smoothness at framerates closer to the refresh rate.

Thanks for the explanations! The way the copies happen is clearer to me now, I didn’t think of different buffer modes and the slow copy to system memory.

Are there any predictions as to when the 1.19 ABI freeze happens?

So I’ve been trying to follow these instuctions, to just set op PRIME, but I can’t get it to work, is there somethig wrong with my config?

https://punpun.xyz/2vgc.txt - xorg.conf
https://punpun.xyz/fs2u.txt - Xorg.0.log
https://punpun.xyz/rcl8.txt - installed packages

First of all, thank you very much for making this beta driver available.
After lots of testing and debugging I finally made this work. So here are my findings to help anyone trying this too.

  1. This won’t work without glamor compiled into the xserver.
    Without glamor (i.e. compiled in, not used), no framebuffer/pixmap will be created and the modesetting driver will die saying ‘No space left on device’

  2. This won’t work with the newer ‘Load module modesetting Section’ in xorg.conf
    With just specifying

Section "Module"
    Load    "modesetting"
EndSection

Section "Device"
   Identifier     "nvidia"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:7:0:0"
    Option         "AllowEmptyInitialConfiguration"
EndSection

Section "ServerFlags"
    Option         "IgnoreABI" "1"
EndSection

The modesetting driver will grab all devices and the nvidia driver gets none.

  1. It only works if modesetting Device/Screen sections in xorg.conf are defined before nvidia Device/Screen sections.
    Otherwise, the modesetting driver just gets initialised and then unloaded.

  2. Glamor has to be disabled.
    As it was before, use AccelMethod none.

Section "Device"
    Identifier     "intel"
    Driver         "modesetting"
    Option         "AccelMethod" "none"
    BusID          "PCI:0:2:0"
EndSection

Hope this helps someone.
Tearing is gone, I’ll report back if I find something unusual.

PS: here’s my working xorg.conf

Section "ServerLayout"
    Identifier     "layout"
    Screen      0  "nvidia" 0 0
    Inactive       "intel"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    Option         "enable" "true"
    Modeline "1280x720_60.00"  74.48  1280 1336 1472 1664  720 721 724 746  -HSync +Vsync
EndSection

Section "Device"
    Identifier     "intel"
    Driver         "modesetting"
    Option         "AccelMethod" "none"
    BusID          "PCI:0:2:0"
EndSection

Section "Device"
    Identifier     "nvidia"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:7:0:0"
    Option         "AllowEmptyInitialConfiguration"
EndSection

Section "Screen"
    Identifier     "nvidia"
    Device         "nvidia"
    Monitor        "Monitor0"
    SubSection     "Display"
        Virtual     1920 1080
        Modes      "1920x1080" "1280x720_60" "800x600"
    EndSubSection
EndSection

Section "ServerFlags"
    Option         "IgnoreABI" "1"
EndSection

PPS:
compiled packages:
libdrm-2.4.70 <–probably not needed but doesn’t hurt either
libXfont2-2.0.1
xf86-video-intel-2.99.917_p20160812 <–just needed to switch back to plain iGPU
xf86-input-evdev-2.10.3
xorg-server-1.18.99.1_p2a79be9

untouched/already installed
mesa-11.0.6
kernel-4.6.7

configure options for xorg-server (for systemd):
–prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --disable-dependency-tracking --disable-silent-rules --libdir=/usr/lib64 --docdir=/usr/share/doc/xorg-server-1.18.99.1_p2a79be9 --enable-shared --disable-static --enable-ipv6 --disable-dmx --enable-glamor --enable-kdrive --enable-kdrive-kbd --enable-kdrive-mouse --enable-kdrive-evdev --enable-install-setuid --disable-tslib --disable-libunwind --disable-xwayland --enable-record --enable-xfree86-utils --enable-dri --enable-dri2 --enable-glx --disable-xephyr --disable-xnest --enable-xorg --disable-xvfb --enable-config-udev --without-doxygen --without-xmlto --with-systemd-daemon --enable-systemd-logind --enable-libdrm --sysconfdir=/etc/X11 --localstatedir=/var --with-fontrootdir=/usr/share/fonts --with-xkb-output=/var/lib/xkb --disable-config-hal --disable-linux-acpi --without-dtrace --without-fop --with-os-vendor=Gentoo --with-sha1=libcrypto

  1. It only works if modesetting Device/Screen sections in xorg.conf are defined before nvidia Device/Screen sections.
    Otherwise, the modesetting driver just gets initialised and then unloaded.

This is the tip that made my PRIME work after hours upon hours of changing configs!
Thanks

Yikes, thanks for tracking this detail down. This will help focus on what needs to be fixed in the server’s configuration code to make this easier.

I did all steps in this tread and stuck at “caught signal 11” message. Part of the log above is the same (not ‘backtrace’).

Using Xserver 1.18.99.1 (git 2a79be9) and nvidia-370 package from Launchpad’s ppa repository.
Fresh Lubuntu 16.04.1.
Tried xorg.conf from post #24.

Can anybody help me if I’ll post more logs?

And thanks.

@YStar
Please post your configure options and your gcc version. I’ll post mine when I find some time. Will have to extract them from the Gentoo ebuild I made.

I built the X server according to this guide: https://www.x.org/wiki/Building_the_X_Window_System/
And by default ‘build.sh’ compiling this list of modules (module-list.txt):
http://pastebin.com/s1CQfkjZ

GCC version: 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2)

@YStar
Erm, looking at the provided modules.txt you’re building some real old xserver-1.12.2 and old modules, not anything from git.

Always a small hickup at start with vanilla kernel 4.6.7 but no noticeable impact.

[ 8.500690] ------------[ cut here ]------------
[ 8.500698] WARNING: CPU: 3 PID: 1344 at kernel/softirq.c:150 __local_bh_enable_ip+0x61/0x90
[ 8.500699] Modules linked in: bnep nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) uvcvideo btusb btrtl btbcm videobuf2_vmalloc btintel videobuf2_memops bluetooth videobuf2_v4l2 iwlmvm videobuf2_core x86_pkg_temp_thermal iwlwifi r8169 efivarfs
[ 8.500710] CPU: 3 PID: 1344 Comm: Xorg Tainted: P U O 4.6.7 #4
[ 8.500711] Hardware name: LENOVO 20287/VIUU4, BIOS 8DCN40WW 10/24/2014
[ 8.500712] 0000000000000000 ffff8802409ef950 ffffffff81516c78 0000000000000000
[ 8.500714] 0000000000000000 ffff8802409ef990 ffffffff81056c6c 0000009600000000
[ 8.500715] 0000000000000200 0000000000000000 ffff88023bae2e48 0000000000000000
[ 8.500717] Call Trace:
[ 8.500722] [] dump_stack+0x4d/0x65
[ 8.500724] [] __warn+0xcc/0xf0
[ 8.500725] [] warn_slowpath_null+0x18/0x20
[ 8.500727] [] __local_bh_enable_ip+0x61/0x90
[ 8.500730] [] _raw_spin_unlock_bh+0x15/0x20
[ 8.500748] [] nvkms_init_timer+0xea/0x110 [nvidia_modeset]
[ 8.500760] [] ? _nv001936kms+0x60/0x60 [nvidia_modeset]
[ 8.500765] [] nvkms_alloc_timer+0x44/0x60 [nvidia_modeset]
[ 8.500775] [] _nv001936kms+0x47/0x60 [nvidia_modeset]
[ 8.500777] [] ? nvidia_drm_gem_prime_fence_op_enable_signaling+0x6e/0x110 [nvidia_drm]
[ 8.500780] [] fence_default_wait+0x91/0x230
[ 8.500783] [] ? ilk_compute_pipe_wm+0x383/0x490
[ 8.500785] [] nvidia_drm_gem_prime_fence_op_wait+0x23/0x30 [nvidia_drm]
[ 8.500786] [] fence_wait_timeout+0x7d/0x160
[ 8.500787] [] reservation_object_wait_timeout_rcu+0x226/0x2a0
[ 8.500790] [] intel_prepare_plane_fb+0xea/0x2e0
[ 8.500791] [] ? mutex_lock+0xd/0x30
[ 8.500794] [] drm_atomic_helper_prepare_planes+0x4b/0xc0
[ 8.500795] [] intel_atomic_commit+0x298/0x1570
[ 8.500798] [] ? drm_atomic_set_crtc_for_connector+0x52/0xe0
[ 8.500799] [] drm_atomic_commit+0x32/0x50
[ 8.500801] [] drm_atomic_helper_set_config+0x76/0xb0
[ 8.500803] [] drm_mode_set_config_internal+0x5d/0xf0
[ 8.500805] [] drm_mode_setcrtc+0x183/0x4d0
[ 8.500807] [] drm_ioctl+0x13d/0x560
[ 8.500808] [] ? drm_mode_setplane+0x1b0/0x1b0
[ 8.500811] [] ? selinux_file_ioctl+0x116/0x1f0
[ 8.500813] [] do_vfs_ioctl+0x8b/0x590
[ 8.500814] [] SyS_ioctl+0x74/0x80
[ 8.500816] [] entry_SYSCALL_64_fastpath+0x17/0x93
[ 8.500817] —[ end trace fedfeb7914e5deea ]—

Edit: this happens after:
[ 6.642138] [drm:intel_dump_crtc_timings] crtc timings: 151600 1920 2010 2070 2220 1080 1086 1095 1138, type: 0x48 flags: 0xa
[ 6.642139] [drm:intel_dump_pipe_config] port clock: 270000
[ 6.642139] [drm:intel_dump_pipe_config] pipe src size: 1920x1080
[ 6.642140] [drm:intel_dump_pipe_config] num_scalers: 0, scaler_users: 0x0, scaler_id: 0
[ 6.642141] [drm:intel_dump_pipe_config] gmch pfit: control: 0x00000000, ratios: 0x00000000, lvds border: 0x00000000
[ 6.642142] [drm:intel_dump_pipe_config] pch pfit: pos: 0x00000000, size: 0x00000000, disabled
[ 6.642143] [drm:intel_dump_pipe_config] ips: 0
[ 6.642144] [drm:intel_dump_pipe_config] double wide: 0
[ 6.642145] [drm:intel_dump_pipe_config] ddi_pll_sel: 536870912; dpll_hw_state: wrpll: 0x0 spll: 0x0
[ 6.642145] [drm:intel_dump_pipe_config] planes on this crtc
[ 6.642147] [drm:intel_dump_pipe_config] STANDARD PLANE:23 plane: 0.0 idx: 0 disabled, scaler_id = 0
[ 6.642148] [drm:intel_dump_pipe_config] CURSOR PLANE:25 plane: 0.1 idx: 1 disabled, scaler_id = 0
[ 6.642149] [drm:intel_dump_pipe_config] STANDARD PLANE:27 plane: 0.1 idx: 2 disabled, scaler_id = 0
[ 6.642161] ------------[ cut here ]------------
[ 6.642166] WARNING: CPU: 3 PID: 1355 at kernel/softirq.c:150 __local_bh_enable_ip+0x61/0x90

Oooh, this is great.
xrandr --output eDP-1 --set “PRIME Synchronization” 0
Tearing there
xrandr --output eDP-1 --set “PRIME Synchronization” 1
Tearing gone.

Regarding power management, I noticed that, in its current form, the NVIDIA driver won’t actually turn the dGPU off when it’s set to use the iGPU. Instead, it leaves the dGPU on but idling while the iGPU does the work.

When testing on my Vostro 5470 with an onboard GT 740M, it consumed around 10W when idling with the standard PRIME solution while it consumed just about 6W when using bbswitch to turn the dGPU off (after unloading the NVIDIA modules). I wonder why the default configuration doesn’t do that, since it proved to be quite reliable for me to automatically disable the dGPU when using the iGPU, which almost doubled my battery life in common usage cenarios (such as surfing the internet and programming). It hasn’t caused any issues with suspend and hibernation and I even managed to turn the dGPU back on to run CUDA programs, etc.

My code is based around the prime-indicator applet and can be found here: http://github.com/andrebrait/prime-indicator

A similar approach could be used by the oficial drivers so we would still get optimal battery usage when not using the dGPU.

Is the ABI frozen yet? 1.19RC1 has been out for a while.

No, the ABI hasn’t officially been frozen and I’ve seen talk of another ABI break to fix some broken hardware cursor changes.

New 375 beta is out, does it have prime sync capabilities? If so what xorg commit is it built on?
Having some minor regressions with 270.28 over .23 I’d like to try that.

Yes, it and all drivers going forward should have PRIME Sync capabilities. As of this release it’s still built against commit 2a79be9, as shown here: http://us.download.nvidia.com/XFree86/Linux-x86_64/375.10/README/randr14.html . If and when the commit changes, it will be reflected on that page in the respective README.

Ok, thanks, this help a lot.

  • configuration (hw)
    iGPU (intel)
    dGPU (nvidia)
    two monitors attached to each card

  • why do i need PRIME?
    to run kmscon. kmscon doesn’t run on nvidia blob but it does on intel’s.

  • what is the problem now?

kmscon runs on tty1-5 and X on tty6 with PRIME synchronisation. problem is that monitors are mirrored. i have tried TwinView but it doesn’t seem to work. here is my xorg.conf:

Section "ServerLayout"
    Identifier     "layout"
    Screen      0 "nvidia"
    Screen      1 "intel" rightOf "nvidia"
    Inactive "intel"
EndSection

Section "Files"
EndSection

Section "InputDevice"
    # generated from data in "/etc/conf.d/gpm"
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol"
    Option         "Device" "/dev/input/mice"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"
    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Samsung SMBX2335"
    HorizSync       30.0 - 81.0
    VertRefresh     56.0 - 75.0
    Option         "DPMS"
EndSection

Section "Monitor"
    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor1"
    VendorName     "Unknown"
    ModelName      "Samsung SMBX2335"
    HorizSync       30.0 - 81.0
    VertRefresh     56.0 - 75.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "nvidia"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID "PCI:1:0:0"
EndSection

Section "Screen"
    Identifier     "nvidia"
    Monitor        "Monitor0"
    Device         "nvidia"
    DefaultDepth    24
    Option "AllowEmptyInitialConfiguration"
    Option "TwinView" "true"
EndSection

Section "Device"
    Identifier     "intel"
    Driver         "modesetting"
EndSection

Section "Screen"
    Identifier     "intel"
    Monitor        "Monitor1"
    Device         "intel"
    DefaultDepth    24
    Option "TwinView" "true"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

any idea?

update: rearranged config sections for better reading:

Section "InputDevice"
    # generated from data in "/etc/conf.d/gpm"
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol"
    Option         "Device" "/dev/input/mice"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection
Section "InputDevice"
    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Device"
    Identifier     "nvidia"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID "PCI:1:0:0"
EndSection
Section "Device"
    Identifier     "intel"
    Driver         "modesetting"
EndSection


Section "Monitor"
    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Samsung SMBX2335"
    HorizSync       30.0 - 81.0
    VertRefresh     56.0 - 75.0
    Option         "DPMS"
EndSection
Section "Monitor"
    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor1"
    VendorName     "Unknown"
    ModelName      "Samsung SMBX2335"
    HorizSync       30.0 - 81.0
    VertRefresh     56.0 - 75.0
    Option         "DPMS"
EndSection


Section "Screen"
    Identifier     "nvidia"
    Monitor        "Monitor0"
    Device         "nvidia"
    DefaultDepth    24
    Option "AllowEmptyInitialConfiguration"
    Option "TwinView" "true"
EndSection
Section "Screen"
    Identifier     "intel"
    Monitor        "Monitor1"
    Device         "intel"
    DefaultDepth    24
    Option "TwinView" "true"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "ServerLayout"
    Identifier     "layout"
    Screen      0 "nvidia"
    Screen      1 "intel" rightOf "nvidia"
    Inactive "intel"
EndSection