Dual GPU Configuration on Ubuntu 21.10

Graphics / Linux Linux
1

Hi,

I’m trying to get a dual GPU setup working on Linux as I’m sick of Windows, always have been, but Linux as a daily driver has been tough. Wanted to give it a shot again…

Installing the built in drivers doesn’t seem to work too well, I haven’t personally had the greatest luck with them on Linux, (I’m actually developing a GPU management software for Linux right now as well) thus I usually use ones downloaded from Nvidia.

Installation was fine of course, other than nouveau not getting blacklisted and for some reason stalling after entering the boot drive’s encryption password, not that if it loaded it should’ve been used for anything but console anyhow but as far as I know it’s best blacklisted and generally only causes issues if it loads anywhere.

So, blacklisted and tossed nouveau.modeset line into kernel. Rebooted, X comes up looking decent. I set up all my screens’ (I have 7 total displays currently hooked up; three monitors in typical triple-monitor fashion as my main displays on my desk connected to my secondary card, a RTX 2070 SUPER, two more above those on stands tilted down towards me, a third rotated vertically to the right of the main three - those three connected to my primary card, RTX 3060 - the 7th display is my A/V receiver on a USB-C->HDMI adapter feeding my projector which is disconnected for current experimentation) orientation and during which I notice a few things.

First is that the graphics are LAGGY. Like, 15fps on the ‘secondary’ card (and while I thought this was physically whether the card was primary/secondary causing this, but was actually whether said card was specified as primary or secondary in the xorg.conf) and about 25fps on the primary card when dragging windows around - essentially unbearable.

I also noticed under the X screen configuration in nvidia-settings that I had two sets of screens for the secondary card. The first, enabled set was through PRIME, the second was not. I was able to configure my displays exactly as needed in this mode - the issue was performance.

After some tinkering with the configuration, I managed to setup two X screens, one for each GPU with it’s corresponding monitors under each. After rebooting, the displays I had configured as primary (my “main” three, connected to the physically secondary GPU) were working fine and the graphics were ULTRA-SMOOTH (my displays are only 60hz but still put Windows to shame) … The other three, were active, but had a black background and when the mouse moved over them was the default ‘X’ cursor - X was treating the secondary screen as a second seat with nothing connected! I couldn’t even force apps over to it by exporting it’s DISPLAY number but I could move the mouse to those screens with no issue. One benefit of this setup was that coolbits was finally working for the secondary card and allowing me to manage it’s clocks as it now had a screen attached to it.

I thought OPTIMUS/PRIME was specifically for hybrid GPU setups…? Let me clearly state that I do NOT have an internal GPU of any kind (9700 non-K). My assumption was that it was still using PRIME to somehow aid in display routing so the desktop could be managed all from one GPU while being displayed on both. I would have zero issue with this if the performance hit wasn’t so severe, literally unusable; also not being able to adjust the clocks on the secondary card is a bit of a dealbreaker - one would hope that if this solution is the preferred method that one could still attach a virtual/AllowEmptyInitialConfiguration-style display to the GPU in order to enable coolbits.

What about Xinerama/nvidiaXinerama? is there a way I can configure my xorg.conf to do this properly? I tried a few variations of configs regarding that without any luck but I was thinking perhaps this is the best way forward…?

I wonder also if this is particularly an Ubuntu-related issue? or at least a more modern-Linux-related issue? I know typically in the past that multiple displays would be exposed to X as separate screens and Xinerama could be used otherwise; even a lot of not THAT old threads show configuring xorg.conf this way but it seems that things have changed now…?

Are we also still only limited to 4-screens per (non-Quadro) GPU on Linux as well? If not (or if there’s a patch like nvenc cap) I could maybe try and use DP-daisychain/splitters to get all my screens going off one card but…I get the feeling that (disgustingly stupid in 2022 - especially when most users would opt for fewer larger displays, unlike me who just can’t afford to upgrade and has a ton of 10+ year old 1080p panels so may as well take the realestate and have a cool looking battlestation) limit is alive and well on Linux as well.

Sort of a TL;DR: What’s the best way to configure multiple GPUs (3+ monitors per GPU) on Linux with being able to arrange/manage screen layout/position properly for all and no (major) performance or UX (dragging windows etc) issues?

The ‘default’ way seemed to push the secondary GPUs screens to the primary GPU with PRIME which caused horrible lagging and was unusable though allowed proper screen orientation configuration but no coolbits for 2nd GPU.

Configuring the secondary GPU as a secondary X screen seemed to be a solution (and enabled coolbits for the 2nd GPU) but left the screens as blank unusable X desktops (that could be moused to).

2

That’s a hardware limitation, current gpus only have 4 display-engines aka heads.
Only way around that are external splitters that pretend to be a 4k monitor and have 4 FHD outputs. Expensive.

Not useful (anymore), see:
https://forums.developer.nvidia.com/t/problem-installing-nvidia-driver-on-red-hat-8-4/195163/9

Forget about this, for two/three/four reasons:

  • hardly working on recent Xservers especially with the nvidia drivers
  • xinerama and compositing doesn’t work, you are limited to using non-compositing WMs
  • GLX might be disabled on one gpu due to being from different generations (Turing+Ampere)
  • many things are done in system memory so overall performance isn’t great.

Yes, but nvidia2nvidia prime was only recently added in driver 470 and using nvidia as an output sink still has bugs and performance issues.
Since your 2070 seems be the bottle-neck, please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.
Maybe I can identify some reason.

Outlook: maybe wayland might make this kind of setup easier and reliable, some day.

3

Not sure that the 2070 specifically is the bottleneck as my primary displays are connected to it and when they’re active alone they work amazingly smooth.

The 2070 is connected via an X1 slot but this has never really affected me on most applications/systems where it’s ever been the case.

I would be willing to run the 2070S with my 2060 - I had actually got the 3060 to replace the 2060, being that I was able to get the 3060 at MSRP in Oct, with the current market I should’ve been able to sell the 2060 and replace it with the 3060, selling the 2060 for what I originally paid for it a few years ago. Not something I’d normally do but in this market it seems like even getting a 2060 for it’s 2-year-old MSRP is a steal.

All moot anyway - my other system is a Mac Pro 3,1; going to the 2060 alone killed my ability to really use OS X as my 970 died at the same time I installed the 2060… and with the 3060 won’t even post if it’s installed at all. If I could get it to post with the 3060 I could swap the 2060 in and have two same gen cards, but I guess that’s not possible.

Seems odd about the 4-display head thing; I thought it was just software; I swear I had 5 displays on one card working one day under Windows (DPx3, HDMIx1, USB-C->HDMIx1 on my 2070S) though one display was mirrored (which was fine as that display was on the projector, though, for that purpose my HDMI matrix boxes can handle mirroring). Seems silly as AFAIK AMD cards, and Quadros don’t have this limitation… In fact the only reason I have a 2nd GPU installed is for running the extra displays, although the extra performance in the few apps that can make use of it is appreciated.

I really want to be done with Windows on my main development ‘battlestation’, I happen to be developing a Linux GPU management program at the moment as well so being able to add my main system as a dev rig would be nice… I don’t even think that a TH2G or a QH2G would really work for me as I’d likely lose things like monitor snapping.

I’ll definitely boot over to Linux in the next it and provide the logs you requested; I guess I’ll reset the Xconfig to the “default” way first, hopefully we can find a way forward. This feels like very basic multi-monitor stuff that should’ve always been supported - I mean, ‘back in the day’ if you wanted multimonitor, you HAD to use two GPUs. The whole thing with every display manager going to a ‘single screen’-type thing I suppose didn’t help keep it easily supported.

If this prime way is only added recently as of 470 - what option was there before that for multi GPU? Just multiple X screens/Xinerama which is pretty much no longer supported anywhere?

I really hope we can get this going; the performance with just one card (the secondary card, 2070S on x1, hooked to my main 3 displays) was droolworthy smooth compared to Windows even. If I can get that kind of performance on all my displays and get most of my main apps setup then I can kick Windows to the curb for the most part.

I suppose this would be even more of a nightmare with different vendor GPUs. I haven’t successfully gotten X to run recognizing both vendor GPUs on my gpu-dev-test rig yet (though that one is technically headless atm); generally the AMD driver crashes with a drm call as if it’s hijacking all the NV drm calls or something.

Help me NV-won-kenobi, you’re my only hope!

4

The 4-head limit is a hardware limit. Under certain circumstances two or even more connectors can share the same head in mirrored mode, sharing the same PLL.
I’d still need the nvidia-bug-report.log, though.

5

Seems like this user has a similar setup and got it working (although, same GPUs and have a bridge…)

I reinstalled the driver and let it set xorg to “default” settings; this is the config where I am able to setup the desktop layout as needed, but performance is terrible on the GPU defined as secondary (doesn’t matter which GPU is defined as secondary, the performance is the same on both when they are configured as secondary) with the primary GPU only being a bit better (but no where as performant as when using a single GPU or using separate X screens on both [albeit, nothing is displayed on the secondary screen], or even using whatever ‘default’ driver it used when I reinstalled the nvidia driver and loaded X before rebooting, thus the new driver wasn’t signed, thus not loaded - and noveau is blacklisted so I’m not sure what it was using but it was also fast, though it mirrored across all screens)

Would the solution shown there help me at all…? Currently with the ‘default’ config it has 6 screens on a single X display; the only issue is performance. I could even deal with slightly degraded performance on a secondary card and use that for the top displays but this is excessive. Even the mouse cursor is choppy on the secondary GPU - again - it doesn’t seem to matter WHICH GPU is set to primary or secondary which should rule out anything like different bus speeds between cards (IE, if one card was always slow that could make sense but the slow card is just whichever is not set as primary/Device0 in the xorg config), just whichever is secondary runs much slower and the primary is also slower but not as bad as the secondary.

Tried playing with settings again today with no real luck so far. What about PRIME Synchronization? Would that maybe work to sync things up/help? Any type of Mosaic? Sorry; unfortunately it seems like a ton of documentation is outdated.

Also, tried something like glxgears … it runs at 60fps on all screens ?? Even when splayed across 3 screens? But window dragging, mouse movement etc is slow still. vkcube is not as bad as window dragging but doesn’t feel perfectly smooth, I don’t see a fps for it? Seems slightly smoother when moved to primary gpu screen maybe… if I run vkcube with --gpu_number 1, it says “WARNING: lavapipe is not a conformant vulkan implementation” but the cube appears to rotate much much faster, like 4x the rotation speed?

I’ve attached the requested log; it says processing still - hopefully it’ll work.

Processing: nvidia-bug-report.log.gz…

6

The virus scanner is stuck again, the file will never be available. please try unzipping and uploading it to another post. Or maybe try to attach it to a PM.
In the thread you referenced, the user is using the same setup like you. It’s just the problem that while it works fine for most users, some have the same performance issue like you. Deeper reason yet unknown to me. I have the suspicion this isn’t really a driver problem but one of the WM compositor, maybe not getting the right vsync signal, thus slowing the whole desktop down. Which DE are you using?

7

nvidia-bug-report.log.gz (643.7 KB)

I’m using gnome; whatever Ubuntu-Desktop installs by default. Just did a regular install of 21.10 and then installed GPU drivers. Haven’t done anything else yet. I was considering trying to get Compiz working somehow for some nice flashy effects; I saw someone did it not too long ago in a video on Ubuntu… I feel like we’ve gone backwards maybe? Like Linux used to have better support for random ‘advanced’ setups like this…? I dunno… I didn’t even have problems with this type of setup on OS X with my Mac Pro…but that went out the Window when I swapped the 970 for a 2060…no more OSX…then I went to swap that 2060 with the 3060 and it wouldn’t even post!

I do have a 1660TI (though, it seems to not want to boot on EFI machines? works fine in my legacy machine…) and a 2060 I could potentially test with, 2060+2070S would both be turing at least but I’d prefer to keep the 3060/2070S as the 12gb vram on the 3060 comes in handy with Blender and stuff.

I did notice that the GPU on x1 was showing a bit high PCIe load when moving windows; obviously if it has to pipe the display buffer over PCIe for this to work then I get some degradation but this is extreme; and again the GPU not set as primary in xorg suffers much worse, regardless of which GPU or slot.

Also of course, I have zero issues on Windows relating to the bandwidth; no performance hit in mining, video enc/deenc, gaming, 3D (blender, solidworks, fusion360, etc) or the desktop - everything just works pretty flawlessly, even when Windows span multiple displays across multiple cards… I do have a rather random assortment of screens; looking (at least under Linux) it seems like not all support 60hz exactly; though under Windows I’m pretty sure they all run at 60hz no problem; actually had my main displays running at 65hz I think.

I suppose as well I likely can’t use anything like NVBroadcast under Linux either, even with something like Wine?

8

There’s something wrong with the 3060, it stays at minimum clocks, also limiting the pcie bus speed to minimum so the bus gets clogged.
Does it clock up if you use nvidia-settings to set it to “prefer maximum performance” in the powermizer pane? The 2070 seems to work fine, using medium clocks to adapt to the needed pcie speed.

9

The 3060 is on a full 3.0 x16 … I believe it does clock up but I’ll have to double check. I have no issues clocking the 1660, it runs on a similar Linux system with two AMD RX570s… While I saw high-ish usage on the 2070S for bus (upto 60%-ish when dragging windows) the 3060 bus usage never exceeded ~6-7%… For the record GPU-Z on windows shows the 3060 as x16 4.0 capable running at x16 3.0 and the 2070S running at x1 3.0.

I did notice that I was unable to set fan speeds in the control panel; I didn’t try from the console yet. This is with downloaded nvidia drivers; I also tried with the built in ubuntu drivers originally and I believe I got the same result.

10

Fan control and overclocking only works if X runs as root.

11

The powermizer pane displays Current PCIe Link Speed, seems to be only 2,5 GT/s. Set preferred mode to maximum, then check if it changes to 8 GT/s. Check if performance is correct.

12

I had tried setting it to maximum before with no difference really; maybe a minor improvement but nothing crazy… does it need to be run as root to set preferred mode? You’d think nvidia-settings would ask/acquire permissions if needed or at least present an error; well, it does on console obviously but in the GUI just clicking something and having it do nothing is not a great UX/UI response, heh…

I’ll double check the PCI speeds as well next time I boot over… any other suggestions on things to try?

13

Setting the preferred mode doesn’t require root, works as user instantly. Only the coolbits extras fanctrl/overclocking require root.

14

Did you ever get this working correctly? I’m currently in a similar situation - 4 monitors on 1 GPU (3080ti) and 2 on another (4060ti) and have it to the point where I can run a 2nd window manager on screen 0.1, but I’d much rather have all 6 cohesively managed by KDE.

Sounds like Prime might be an option - going to install that and see if it helps…