PRIME Render Offload doesn't work on Ubuntu 18.04 with the latest driver

I have installed the Linux driver and followed instructions in https://download.nvidia.com/XFree86/Linux-x86_64/440.44/README/primerenderoffload.html to set up the PRIME Render Offload. I have a problem that nvidia-drm is not loaded:

lsmod | grep nvidia
nvidia              20361216  12
ipmi_msghandler       102400  2 ipmi_devintf,nvidia

However,

$nvidia-smi
Tue Dec 24 00:50:16 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1650    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   44C    P8     2W /  N/A |      0MiB /  3911MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
$ xrandr --listproviders
Providers: number : 1
Provider 0: id: 0x43 cap: 0x9, Source Output, Sink Offload crtcs: 3 outputs: 1 associated providers: 0 name:modesetting

If I try to load nvidia-drm with sudo modprobe nvidia-drm, sometimes the system freezes completely; otherwise I get two identical modesetting lines in xrandr --listproviders. If I check my Xorg.0.log I have

[    38.331] (==) Using config file: "/etc/X11/xorg.conf"
[    38.331] (==) Using system config directory "/usr/share/X11/xorg.conf.d"
[    38.332] (==) ServerLayout "layout"
[    38.332] (**) |-->Screen "iGPU" (0)
[    38.332] (**) |   |-->Monitor "<default monitor>"
[    38.332] (**) |   |-->Device "iGPU"
[    38.332] (**) |   |-->GPUDevice "dGPU"
[    38.332] (**) |   |-->GPUDevice "iGPU"
[    38.332] (**) |   |-->GPUDevice "dGPU"
[    38.332] (==) No monitor specified for screen "iGPU".
        Using a default monitor configuration.
...
[    38.337] (II) LoadModule: "nvidia"
[    38.337] (II) Loading /usr/lib/xorg/modules/drivers/nvidia_drv.so
[    38.341] (II) Module nvidia: vendor="NVIDIA Corporation"
[    38.341]    compiled for 1.6.99.901, module version = 1.0.0
[    38.341]    Module class: X.Org Video Driver
[    38.342] (II) modesetting: Driver for Modesetting Kernel Drivers: kms
[    38.342] (II) NVIDIA dlloader X Driver  440.44  Sun Dec  8 03:38:31 UTC 2019
[    38.342] (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
[    38.342] xf86EnableIOPorts: failed to set IOPL for I/O (Operation not permitted)
[    38.342] (II) modeset(0): using drv /dev/dri/card0
[    38.342] (WW) Falling back to old probe method for modesetting
...
[    38.344] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card support
[    38.344] (EE) Screen 1 deleted because of no matching config section.
[    38.344] (II) UnloadModule: "nvidia"

My /etc/X11/xorg.conf is as follows

Section "ServerLayout"
        Identifier "layout"
        Screen 0 "iGPU"
        Option "AllowNVIDIAGPUScreens"
EndSection

Section "Device"
        Identifier "iGPU"
        Driver "modesetting"
        BusID "PCI:0:2:0"
EndSection

Section "Screen"
        Identifier "iGPU"
        Device "iGPU"
EndSection

Section "Device"
        Identifier "dGPU"
        Driver "nvidia"
        BusID "PCI:1:0:0"
EndSection

There seem to be some additional config files interfering.
Please run nvidia-bug-report.sh as root and attach the resulting .gz file to your post. Hovering the mouse over an existing post of yours will reveal a paperclip icon.
https://devtalk.nvidia.com/default/topic/1043347/announcements/attaching-files-to-forum-topics-posts/

I have attached the nvidia-bug-report.log.gz to the original post. Here are the config files in /usr/share/X11/xorg.conf.d:

ls -l /usr/share/X11/xorg.conf.d
total 24
-rw-r--r-- 1 root root   92 May 29  2019 10-amdgpu.conf
-rw-r--r-- 1 root root 1350 May  2  2019 10-quirks.conf
-rw-r--r-- 1 root root   92 May 29  2019 10-radeon.conf
-rw-r--r-- 1 root root  945 Nov 27  2018 40-libinput.conf
-rw-r--r-- 1 root root 3025 Nov 27  2018 70-wacom.conf
-rw-r--r-- 1 root root  340 Dec 24 00:15 xorg.conf

Here xorg.conf has the same content as /etc/X11/xorg.conf in the first post.

Looks like you had the same config twice, once in /etc/X11/xorg.conf and one in /usr/share/X11/xorg.conf.d/xorg.conf so all gpus got loaded twice:

[    38.332] (**) |   |-->Device "iGPU"
[    38.332] (**) |   |-->GPUDevice "dGPU"
[    38.332] (**) |   |-->GPUDevice "iGPU"
[    38.332] (**) |   |-->GPUDevice "dGPU"

Please remove the one in /usr/share/X11/xorg.conf.d/xorg.conf
Furthermore, render offload needs a patched Xserver, I don’t think stock Ubuntu 18.04 provides that yet, or did you use the Xserver from nvidia ppa? http://download.nvidia.com/XFree86/Linux-x86_64/435.17/README/primerenderoffload.html

I removed the second xorg.conf and now gpus only appear once (see updated bug log attached). I have patched the Xserver. But I still have a problem - I ran sudo modprobe nvidia-drm and now get

xrandr --listproviders
Providers: number : 2
Provider 0: id: 0x43 cap: 0x9, Source Output, Sink Offload crtcs: 3 outputs: 1 associated providers: 1 name:modesetting
Provider 1: id: 0x1ca cap: 0x2, Sink Output crtcs: 4 outputs: 3 associated providers: 1 name:modesetting

Config looks correct, I suspect this is a known timing problem, the nvidia driver gets loaded too late so the Xserver is already up. Please try embedding the nvidia driver into the initrd so they’re available as early as possible:
https://askubuntu.com/questions/676707/how-to-load-a-module-in-initrd

I added the nvidia drivers (nvidia-drm, nvidia-modeset, nvidia) to the modules file in the initrd but they are still being unloaded:

dmesg | grep nvidia
[    2.177183] nvidia: loading out-of-tree module taints kernel.
[    2.177190] nvidia: module license 'NVIDIA' taints kernel.
[    2.208640] nvidia-nvlink: Nvlink Core is being initialized, major device number 236
[    2.208878] nvidia 0000:01:00.0: enabling device (0006 -> 0007)
[    2.209058] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    2.261647] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  440.44  Sun Dec  8 03:29:48 UTC 2019
[    2.263391] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[    3.019264] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1
[   35.630236] [drm] [nvidia-drm] [GPU ID 0x00000100] Unloading driver
[   35.745213] nvidia-modeset: Unloading
[   35.776614] nvidia-nvlink: Unregistered the Nvlink Core, major device number 236
[   36.883468] nvidia-nvlink: Nvlink Core is being initialized, major device number 236
[   37.167497] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none:owns=none

Could it be related to this in Xorg.0.log?

[    47.109] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card support
[    47.109] (EE) Screen 1 deleted because of no matching config section.

Additional info in /var/log/gpu-manager.log

Looking for nvidia modules in /lib/modules/5.0.0-37-generic/updates/dkms
Looking for amdgpu modules in /lib/modules/5.0.0-37-generic/updates/dkms
Is nvidia loaded? yes
Was nvidia unloaded? no
Is nvidia blacklisted? no
Is intel loaded? yes
Is radeon loaded? no
Is radeon blacklisted? no
Is amdgpu loaded? no
Is amdgpu blacklisted? no
Is amdgpu versioned? no
Is amdgpu pro stack? no
Is nouveau loaded? no
Is nouveau blacklisted? yes
Is nvidia kernel module available? no
Is amdgpu kernel module available? no
Vendor/Device Id: 8086:3e9b
BusID "PCI:0@0:2:0"
Is boot vga? yes
Vendor/Device Id: 10de:1f91
BusID "PCI:1@0:0:0"
Is boot vga? no
Skipping "/dev/dri/card1", driven by "nvidia-drm"
Skipping "/dev/dri/card0", driven by "i915"
Skipping "/dev/dri/card1", driven by "nvidia-drm"
Skipping "/dev/dri/card0", driven by "i915"
Skipping "/dev/dri/card1", driven by "nvidia-drm"
Skipping "/dev/dri/card0", driven by "i915"
Skipping "/dev/dri/card1", driven by "nvidia-drm"
Found "/dev/dri/card0", driven by "i915"
output 0:
	card0-eDP-1
Number of connected outputs for /dev/dri/card0: 1
Does it require offloading? yes
last cards number = 2
Has amd? no
Has intel? yes
Has nvidia? yes
How many cards? 2
Has the system changed? No
Intel IGP detected
Intel hybrid system
Removing /usr/share/X11/xorg.conf.d/11-nvidia-prime.conf
Unloading nvidia-drm with "no" parameters
Unloading nvidia-uvm with "no" parameters
Unloading nvidia-modeset with "no" parameters
Unloading nvidia with "no" parameters
Setting power control to "auto" in /sys/bus/pci/devices/0000:01:00.0/power/control

Please try disabling gpu-manager using kernel parameter
nogpumanager

Thanks, generix, it finally works!

xrandr --listproviders
Providers: number : 2
Provider 0: id: 0x1bc cap: 0x9, Source Output, Sink Offload crtcs: 3 outputs: 1 associated providers: 0 name:modesetting
Provider 1: id: 0x198 cap: 0x0 crtcs: 0 outputs: 0 associated providers: 0 name:NVIDIA-G0
nvidia-smi    
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1650    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   40C    P8     1W /  N/A |     16MiB /  3911MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1847      G   /usr/lib/xorg/Xorg                            14MiB |
+-----------------------------------------------------------------------------+

I have a Xorg process running on the GPU from the bootup (see above), is that correct? I think the overall desktop environment is rendered by Intel’s integrated graphics, nevertheless, because if I run

glxinfo | grep vendor
server glx vendor string: SGI
client glx vendor string: Mesa Project and SGI
OpenGL vendor string: Intel Open Source Technology Center

Thanks for the help, generix. Are you a Nvidia employee? if not, they should hire you in recognition of all the help you’re giving on these forums.