Cannot use second GPU in linux

[NutFoo]

Issue:
Every port on Card0 works just fine
I cannot use any ports on Card1, I get an error from xrandr (See Below)

markm {~} kobaz$ xrandr --listproviders
Providers: number : 2
Provider 0: id: 0x217 cap: 0x1, Source Output crtcs: 4 outputs: 8 associated providers: 1 name:NVIDIA-0
Provider 1: id: 0x43f cap: 0x2, Sink Output crtcs: 4 outputs: 8 associated providers: 1 name:NVIDIA-G0

markm {~} kobaz$ lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P1000] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
04:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P1000] (rev a1)
04:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
markm {~} kobaz$

xrandr --output DP-0 --mode 2560x1440 --pos 4480x0 --rotate normal --output DP-1 --off --output DP-2 --primary --mode 2560x1440 --pos 1920x1440 --rotate normal --output DP-3 --off --output DP-4 --mode 2560x1440 --pos 1920x0 --rotate normal --output DP-5 --off --output DP-6 --mode 2560x1440 --pos 4480x1440 --rotate normal --output DP-7 --off --output DP-1-0 --off --output DP-1-1 --off --output DP-1-2 --off --output DP-1-3 --off --output DP-1-4 --off --output DP-1-5 --mode 1920x1080 --pos 0x360 --rotate normal --output DP-1-6 --off --output DP-1-7 --off
xrandr: Configure crtc 4 failed

markm-new {~} kobaz$ xrandr --auto
xrandr: Configure crtc 4 failed

markm {~} kobaz$ xrandr
Screen 0: minimum 8 x 8, current 5120 x 2880, maximum 32767 x 32767
DP-0 connected 2560x1440+2560+0 (normal left inverted right x axis y axis) 597mm x 336mm
2560x1440 60.00*+ 75.00
2560x1080 59.94 50.00 29.97 25.00
2048x1152 60.00
1920x1440 60.00
1920x1200 59.88
1920x1080 60.00 59.94 50.00 29.97 25.00
1680x1050 59.95
1440x900 59.89
1440x576 50.00
1440x480 59.94
1280x1024 75.02 60.02
1280x720 60.00 59.94 50.00
1024x768 75.03 70.07 60.00
800x600 75.00 72.19 60.32 56.25
720x576 50.00
720x480 59.94
640x480 75.00 72.81 59.94 59.93
DP-1 disconnected (normal left inverted right x axis y axis)
DP-2 connected primary 2560x1440+0+1440 (normal left inverted right x axis y axis) 597mm x 336mm
2560x1440 60.00*+ 75.00
2560x1080 59.94 50.00 29.97 25.00
2048x1152 60.00
1920x1440 60.00
1920x1200 59.88
1920x1080 60.00 59.94 50.00 29.97 25.00
1680x1050 59.95
1440x900 59.89
1440x576 50.00
1440x480 59.94
1280x1024 75.02 60.02
1280x720 60.00 59.94 50.00
1024x768 75.03 70.07 60.00
800x600 75.00 72.19 60.32 56.25
720x576 50.00
720x480 59.94
640x480 75.00 72.81 59.94 59.93
DP-3 disconnected (normal left inverted right x axis y axis)
DP-4 connected 2560x1440+0+0 (normal left inverted right x axis y axis) 597mm x 336mm
2560x1440 60.00*+ 75.00
2560x1080 59.94 50.00 29.97 25.00
2048x1152 60.00
1920x1440 60.00
1920x1200 59.88
1920x1080 60.00 59.94 50.00 29.97 25.00
1680x1050 59.95
1440x900 59.89
1440x576 50.00
1440x480 59.94
1280x1024 75.02 60.02
1280x720 60.00 59.94 50.00
1024x768 75.03 70.07 60.00
800x600 75.00 72.19 60.32 56.25
720x576 50.00
720x480 59.94
640x480 75.00 72.81 59.94 59.93
DP-5 disconnected (normal left inverted right x axis y axis)
DP-6 connected 2560x1440+2560+1440 (normal left inverted right x axis y axis) 597mm x 336mm
2560x1440 60.00*+ 75.00
2560x1080 59.94 50.00 29.97 25.00
2048x1152 60.00
1920x1440 60.00
1920x1200 59.88
1920x1080 60.00 59.94 50.00 29.97 25.00
1680x1050 59.95
1440x900 59.89
1440x576 50.00
1440x480 59.94
1280x1024 75.02 60.02
1280x720 60.00 59.94 50.00
1024x768 75.03 70.07 60.00
800x600 75.00 72.19 60.32 56.25
720x576 50.00
720x480 59.94
640x480 75.00 72.81 59.94 59.93
DP-7 disconnected (normal left inverted right x axis y axis)
DP-1-0 disconnected (normal left inverted right x axis y axis)
DP-1-1 disconnected (normal left inverted right x axis y axis)
DP-1-2 disconnected (normal left inverted right x axis y axis)
DP-1-3 disconnected (normal left inverted right x axis y axis)
DP-1-4 disconnected (normal left inverted right x axis y axis)
DP-1-5 connected (normal left inverted right x axis y axis)
1920x1080 60.00 +
1680x1050 59.95
1440x900 59.89
1280x1024 75.02 60.02
1280x960 60.00
1280x720 60.00
1024x768 75.03 70.07 60.00
800x600 75.00 72.19 60.32 56.25
640x480 75.00 72.81 59.94
DP-1-6 disconnected (normal left inverted right x axis y axis)
DP-1-7 disconnected (normal left inverted right x axis y axis)
1920x1080 (0x226) 148.500MHz +HSync +VSync
h: width 1920 start 2008 end 2052 total 2200 skew 0 clock 67.50KHz
v: height 1080 start 1084 end 1089 total 1125 clock 60.00Hz
1680x1050 (0x22b) 146.250MHz -HSync +VSync
h: width 1680 start 1784 end 1960 total 2240 skew 0 clock 65.29KHz
v: height 1050 start 1053 end 1059 total 1089 clock 59.95Hz
1440x900 (0x22c) 106.500MHz -HSync +VSync
h: width 1440 start 1520 end 1672 total 1904 skew 0 clock 55.93KHz
v: height 900 start 903 end 909 total 934 clock 59.89Hz
1280x1024 (0x22f) 135.000MHz +HSync +VSync
h: width 1280 start 1296 end 1440 total 1688 skew 0 clock 79.98KHz
v: height 1024 start 1025 end 1028 total 1066 clock 75.02Hz
1280x1024 (0x230) 108.000MHz +HSync +VSync
h: width 1280 start 1328 end 1440 total 1688 skew 0 clock 63.98KHz
v: height 1024 start 1025 end 1028 total 1066 clock 60.02Hz
1280x720 (0x231) 74.250MHz +HSync +VSync
h: width 1280 start 1390 end 1430 total 1650 skew 0 clock 45.00KHz
v: height 720 start 725 end 730 total 750 clock 60.00Hz
1024x768 (0x234) 78.750MHz +HSync +VSync
h: width 1024 start 1040 end 1136 total 1312 skew 0 clock 60.02KHz
v: height 768 start 769 end 772 total 800 clock 75.03Hz
1024x768 (0x235) 75.000MHz -HSync -VSync
h: width 1024 start 1048 end 1184 total 1328 skew 0 clock 56.48KHz
v: height 768 start 771 end 777 total 806 clock 70.07Hz
1024x768 (0x236) 65.000MHz -HSync -VSync
h: width 1024 start 1048 end 1184 total 1344 skew 0 clock 48.36KHz
v: height 768 start 771 end 777 total 806 clock 60.00Hz
800x600 (0x237) 49.500MHz +HSync +VSync
h: width 800 start 816 end 896 total 1056 skew 0 clock 46.88KHz
v: height 600 start 601 end 604 total 625 clock 75.00Hz
800x600 (0x238) 50.000MHz +HSync +VSync
h: width 800 start 856 end 976 total 1040 skew 0 clock 48.08KHz
v: height 600 start 637 end 643 total 666 clock 72.19Hz
800x600 (0x239) 40.000MHz +HSync +VSync
h: width 800 start 840 end 968 total 1056 skew 0 clock 37.88KHz
v: height 600 start 601 end 605 total 628 clock 60.32Hz
800x600 (0x23a) 36.000MHz +HSync +VSync
h: width 800 start 824 end 896 total 1024 skew 0 clock 35.16KHz
v: height 600 start 601 end 603 total 625 clock 56.25Hz
640x480 (0x23d) 31.500MHz -HSync -VSync
h: width 640 start 656 end 720 total 840 skew 0 clock 37.50KHz
v: height 480 start 481 end 484 total 500 clock 75.00Hz
640x480 (0x23e) 31.500MHz -HSync -VSync
h: width 640 start 656 end 696 total 832 skew 0 clock 37.86KHz
v: height 480 start 481 end 484 total 520 clock 72.81Hz
640x480 (0x23f) 25.175MHz -HSync -VSync
h: width 640 start 656 end 752 total 800 skew 0 clock 31.47KHz
v: height 480 start 490 end 492 total 525 clock 59.94Hz

[generix]

Please either set kernel parameter
nvidia-drm.modeset=1
or enable Mosaic.
Either or, not both.

[NutFoo]

Spectacular!

This fixed the issue!

Thank you!!!

[NutFoo]

The quick fix for me was the kernel option: nvidia-drm.modeset=1

[NutFoo]

Second question. I’m getting bad draw performance on the second gpu with nvidia-drm.modeset=1
According to xrandr (and the screens themselves) my refresh is at 60hz

The way I’m testing right now is just purely a visual test. If I drag a window around there’s some decent tearing, but when I’m on any of the screens connected to the first gpu, there’s no tearing.

Would Mosaic offer any difference in performance?

[generix]

Which DE are you using?

[NutFoo]

Using xorg/awesomeWM

Here’s some performace testing:

GPU-0 (Main)

glmark2 2023.01

=======================================================
OpenGL Information
GL_VENDOR: NVIDIA Corporation
GL_RENDERER: Quadro P1000/PCIe/SSE2
GL_VERSION: 4.6.0 NVIDIA 520.56.06
Surface Config: buf=32 r=8 g=8 b=8 a=8 depth=24 stencil=0 samples=0
Surface Size: 800x600 windowed

[build] use-vbo=false: FPS: 3281 FrameTime: 0.305 ms
[build] use-vbo=true: FPS: 4051 FrameTime: 0.247 ms
[texture] texture-filter=nearest: FPS: 3909 FrameTime: 0.256 ms
[texture] texture-filter=linear: FPS: 3868 FrameTime: 0.259 ms
[texture] texture-filter=mipmap: FPS: 3232 FrameTime: 0.309 ms
[shading] shading=gouraud: FPS: 2437 FrameTime: 0.410 ms
[shading] shading=blinn-phong-inf: FPS: 3099 FrameTime: 0.323 ms
[shading] shading=phong: FPS: 3828 FrameTime: 0.261 ms
[shading] shading=cel: FPS: 3799 FrameTime: 0.263 ms
[bump] bump-render=high-poly: FPS: 3036 FrameTime: 0.329 ms
[bump] bump-render=normals: FPS: 3800 FrameTime: 0.263 ms
[bump] bump-render=height: FPS: 4132 FrameTime: 0.242 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 3387 FrameTime: 0.295 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 2259 FrameTime: 0.443 ms
[pulsar] light=false:quads=5:texture=false: FPS: 3784 FrameTime: 0.264 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 1657 FrameTime: 0.604 ms
[desktop] effect=shadow:windows=4: FPS: 2640 FrameTime: 0.379 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 1606 FrameTime: 0.623 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 1898 FrameTime: 0.527 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 1633 FrameTime: 0.613 ms
[ideas] speed=duration: FPS: 3553 FrameTime: 0.282 ms
[jellyfish] : FPS: 3193 FrameTime: 0.313 ms
[terrain] : FPS: 535 FrameTime: 1.872 ms
[shadow] : FPS: 2887 FrameTime: 0.346 ms
[refract] : FPS: 1015 FrameTime: 0.986 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 3624 FrameTime: 0.276 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 3405 FrameTime: 0.294 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 4050 FrameTime: 0.247 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 4023 FrameTime: 0.249 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 3891 FrameTime: 0.257 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 3774 FrameTime: 0.265 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 3553 FrameTime: 0.282 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 3510 FrameTime: 0.285 ms

                              glmark2 Score: 3100

=======================================================

GPU-1 (Secondary)

=======================================================
markm {~} kobaz$ glmark2

glmark2 2023.01

=======================================================
OpenGL Information
GL_VENDOR: NVIDIA Corporation
GL_RENDERER: Quadro P1000/PCIe/SSE2
GL_VERSION: 4.6.0 NVIDIA 520.56.06
Surface Config: buf=32 r=8 g=8 b=8 a=8 depth=24 stencil=0 samples=0
Surface Size: 800x600 windowed

[build] use-vbo=false: FPS: 1010 FrameTime: 0.991 ms
[build] use-vbo=true: FPS: 1508 FrameTime: 0.663 ms
[texture] texture-filter=nearest: FPS: 1340 FrameTime: 0.746 ms
[texture] texture-filter=linear: FPS: 1321 FrameTime: 0.757 ms
[texture] texture-filter=mipmap: FPS: 1253 FrameTime: 0.798 ms
[shading] shading=gouraud: FPS: 1277 FrameTime: 0.784 ms
[shading] shading=blinn-phong-inf: FPS: 1332 FrameTime: 0.751 ms
[shading] shading=phong: FPS: 1263 FrameTime: 0.792 ms
[shading] shading=cel: FPS: 1275 FrameTime: 0.785 ms
[bump] bump-render=high-poly: FPS: 895 FrameTime: 1.118 ms
[bump] bump-render=normals: FPS: 1305 FrameTime: 0.767 ms
[bump] bump-render=height: FPS: 1257 FrameTime: 0.796 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 926 FrameTime: 1.081 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 787 FrameTime: 1.271 ms
[pulsar] light=false:quads=5:texture=false: FPS: 1301 FrameTime: 0.769 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 695 FrameTime: 1.439 ms
[desktop] effect=shadow:windows=4: FPS: 853 FrameTime: 1.173 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 722 FrameTime: 1.386 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 808 FrameTime: 1.239 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 728 FrameTime: 1.375 ms
[ideas] speed=duration: FPS: 1333 FrameTime: 0.750 ms
[jellyfish] : FPS: 913 FrameTime: 1.096 ms
[terrain] : FPS: 360 FrameTime: 2.781 ms
[shadow] : FPS: 898 FrameTime: 1.114 ms
[refract] : FPS: 528 FrameTime: 1.895 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 1309 FrameTime: 0.764 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 1286 FrameTime: 0.778 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 1304 FrameTime: 0.767 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 1288 FrameTime: 0.777 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 1256 FrameTime: 0.796 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 1301 FrameTime: 0.769 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 1283 FrameTime: 0.780 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 1321 FrameTime: 0.757 ms

                              glmark2 Score: 1097

=======================================================

[generix]

With awesomewm, you might either check using an additional compositor or switch to mosaic.

[NutFoo]

Thanks for the info.

Currently I’m using:
/usr/bin/xcompmgr -c -l0 -t0 -r0 -o.00

[NutFoo]

Next issue! Oh man, they keep coming.

If I leave my pc for an extended period of time, and it looks like powersave kicks in. My primary gpu ‘loses connection’. I have to go back into randr and re-activate all my screens

I don’t see anything output in dmesg regarding this.

[generix]

Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.

[NutFoo]

Here is nvidia-bug-report.sh running on a fresh boot prior to any issues.
I will generate a new one when I see the next issue.

https://file.io/YBWxXf6uWx0s

[NutFoo]

Also… I tried using BaseMosaic and I get no X display.

Here’s the config and the log
https://file.io/UXuaonPao2OZ

Thanks!

[generix]

You’re running an outdated driver with a bug so you end up with an Xid 32. Please update your driver.
The BaseMosaic config is wrong, it only needs a very short xorg.conf:

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:1:0:0"
    Option        "BaseMosaic"   "TRUE"
EndSection

nvidia-drm.modeset=1 needs to be unset for BaseMosaic.

[NutFoo]

Hi

I updated the nvdiia driver to NVIDIA-Linux-x86_64-525.105.17

I’ve tried your recommended X config.

Now with BaseMosaic I have 4 outputs working on GPU0 but nothing on GPU1

GPU1 will not ‘turn on’ with Mosaic and nvidia-drm.modeset=0

This form will now allow upload of screenshots so here’s a link.

WIth nvidia-drm.modeset=1 and Mosaic turned off my output looks like this:
DP-1-6 DP-4 DP-0
DP-1-4 DP-2 DP-6

But with mosaic off i have really bad performance on DP-1-6 and DP-1-4

[NutFoo]

Currrent X log with Mosaic turned on

https://file.io/ie8UrN5wjbjK

[NutFoo]

nvidia-drm.modeset=1 and BaseMosaic off

[aplattner]

This configuration has Mosaic disabled:

[  1031.970] (**) NVIDIA(0): Option "Stereo" "0"
[  1031.970] (**) NVIDIA(0): Option "nvidiaXineramaInfoOrder" "DFP-0"
[  1031.970] (**) NVIDIA(0): Option "SLI" "Off"
[  1031.970] (**) NVIDIA(0): Option "MultiGPU" "Off"
[  1031.971] (**) NVIDIA(0): Option "BaseMosaic" "off"
[  1031.971] (**) NVIDIA(0): Stereo disabled by request
[  1031.971] (**) NVIDIA(0): NVIDIA SLI disabled.
[  1031.971] (**) NVIDIA(0): NVIDIA Multi-GPU disabled.

[NutFoo]

Correct. I described this in my series of posts.

[NutFoo]

There are two situations here.

One: Mosaic off, Using randr for screen layout. Everything works initially. Performance is awful on the second gpu, And I randomly lose screens if I’m away from my desk for any period of time. Sometimes DP-2 will be disconnected/inactive. Sometimes DP-2 AND DP-6 will be disconnected/inactive. Other cases DP-1-6, DP-4, DP-2, DP-0, and DP-6 will all be disconnected/inactive and DP-1-4 is the only one still active. I have to re-run randr to resetup all my screens and ‘re-activate’ all the outputs. This is very much a pain because now i have to move sometimes hundreds of windows back to which screens/desktops they are supposed to be on because all the windows fall back to the primary workspace because everything is reset.

Two: Mosaic on
Screen layout seems to be impossible to get all 6 screens to run at the same time.