Dual GPU Configuration on Ubuntu 21.10

Hi,

I’m trying to get a dual GPU setup working on Linux as I’m sick of Windows, always have been, but Linux as a daily driver has been tough. Wanted to give it a shot again…

Installing the built in drivers doesn’t seem to work too well, I haven’t personally had the greatest luck with them on Linux, (I’m actually developing a GPU management software for Linux right now as well) thus I usually use ones downloaded from Nvidia.

Installation was fine of course, other than nouveau not getting blacklisted and for some reason stalling after entering the boot drive’s encryption password, not that if it loaded it should’ve been used for anything but console anyhow but as far as I know it’s best blacklisted and generally only causes issues if it loads anywhere.

So, blacklisted and tossed nouveau.modeset line into kernel. Rebooted, X comes up looking decent. I set up all my screens’ (I have 7 total displays currently hooked up; three monitors in typical triple-monitor fashion as my main displays on my desk connected to my secondary card, a RTX 2070 SUPER, two more above those on stands tilted down towards me, a third rotated vertically to the right of the main three - those three connected to my primary card, RTX 3060 - the 7th display is my A/V receiver on a USB-C->HDMI adapter feeding my projector which is disconnected for current experimentation) orientation and during which I notice a few things.

First is that the graphics are LAGGY. Like, 15fps on the ‘secondary’ card (and while I thought this was physically whether the card was primary/secondary causing this, but was actually whether said card was specified as primary or secondary in the xorg.conf) and about 25fps on the primary card when dragging windows around - essentially unbearable.

I also noticed under the X screen configuration in nvidia-settings that I had two sets of screens for the secondary card. The first, enabled set was through PRIME, the second was not. I was able to configure my displays exactly as needed in this mode - the issue was performance.

After some tinkering with the configuration, I managed to setup two X screens, one for each GPU with it’s corresponding monitors under each. After rebooting, the displays I had configured as primary (my “main” three, connected to the physically secondary GPU) were working fine and the graphics were ULTRA-SMOOTH (my displays are only 60hz but still put Windows to shame) … The other three, were active, but had a black background and when the mouse moved over them was the default ‘X’ cursor - X was treating the secondary screen as a second seat with nothing connected! I couldn’t even force apps over to it by exporting it’s DISPLAY number but I could move the mouse to those screens with no issue. One benefit of this setup was that coolbits was finally working for the secondary card and allowing me to manage it’s clocks as it now had a screen attached to it.

I thought OPTIMUS/PRIME was specifically for hybrid GPU setups…? Let me clearly state that I do NOT have an internal GPU of any kind (9700 non-K). My assumption was that it was still using PRIME to somehow aid in display routing so the desktop could be managed all from one GPU while being displayed on both. I would have zero issue with this if the performance hit wasn’t so severe, literally unusable; also not being able to adjust the clocks on the secondary card is a bit of a dealbreaker - one would hope that if this solution is the preferred method that one could still attach a virtual/AllowEmptyInitialConfiguration-style display to the GPU in order to enable coolbits.

What about Xinerama/nvidiaXinerama? is there a way I can configure my xorg.conf to do this properly? I tried a few variations of configs regarding that without any luck but I was thinking perhaps this is the best way forward…?

I wonder also if this is particularly an Ubuntu-related issue? or at least a more modern-Linux-related issue? I know typically in the past that multiple displays would be exposed to X as separate screens and Xinerama could be used otherwise; even a lot of not THAT old threads show configuring xorg.conf this way but it seems that things have changed now…?

Are we also still only limited to 4-screens per (non-Quadro) GPU on Linux as well? If not (or if there’s a patch like nvenc cap) I could maybe try and use DP-daisychain/splitters to get all my screens going off one card but…I get the feeling that (disgustingly stupid in 2022 - especially when most users would opt for fewer larger displays, unlike me who just can’t afford to upgrade and has a ton of 10+ year old 1080p panels so may as well take the realestate and have a cool looking battlestation) limit is alive and well on Linux as well.

Sort of a TL;DR: What’s the best way to configure multiple GPUs (3+ monitors per GPU) on Linux with being able to arrange/manage screen layout/position properly for all and no (major) performance or UX (dragging windows etc) issues?

The ‘default’ way seemed to push the secondary GPUs screens to the primary GPU with PRIME which caused horrible lagging and was unusable though allowed proper screen orientation configuration but no coolbits for 2nd GPU.

Configuring the secondary GPU as a secondary X screen seemed to be a solution (and enabled coolbits for the 2nd GPU) but left the screens as blank unusable X desktops (that could be moused to).

That’s a hardware limitation, current gpus only have 4 display-engines aka heads.
Only way around that are external splitters that pretend to be a 4k monitor and have 4 FHD outputs. Expensive.

Not useful (anymore), see:
https://forums.developer.nvidia.com/t/problem-installing-nvidia-driver-on-red-hat-8-4/195163/9

Forget about this, for two/three/four reasons:

  • hardly working on recent Xservers especially with the nvidia drivers
  • xinerama and compositing doesn’t work, you are limited to using non-compositing WMs
  • GLX might be disabled on one gpu due to being from different generations (Turing+Ampere)
  • many things are done in system memory so overall performance isn’t great.

Yes, but nvidia2nvidia prime was only recently added in driver 470 and using nvidia as an output sink still has bugs and performance issues.
Since your 2070 seems be the bottle-neck, please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.
Maybe I can identify some reason.

Outlook: maybe wayland might make this kind of setup easier and reliable, some day.

Not sure that the 2070 specifically is the bottleneck as my primary displays are connected to it and when they’re active alone they work amazingly smooth.

The 2070 is connected via an X1 slot but this has never really affected me on most applications/systems where it’s ever been the case.

I would be willing to run the 2070S with my 2060 - I had actually got the 3060 to replace the 2060, being that I was able to get the 3060 at MSRP in Oct, with the current market I should’ve been able to sell the 2060 and replace it with the 3060, selling the 2060 for what I originally paid for it a few years ago. Not something I’d normally do but in this market it seems like even getting a 2060 for it’s 2-year-old MSRP is a steal.

All moot anyway - my other system is a Mac Pro 3,1; going to the 2060 alone killed my ability to really use OS X as my 970 died at the same time I installed the 2060… and with the 3060 won’t even post if it’s installed at all. If I could get it to post with the 3060 I could swap the 2060 in and have two same gen cards, but I guess that’s not possible.

Seems odd about the 4-display head thing; I thought it was just software; I swear I had 5 displays on one card working one day under Windows (DPx3, HDMIx1, USB-C->HDMIx1 on my 2070S) though one display was mirrored (which was fine as that display was on the projector, though, for that purpose my HDMI matrix boxes can handle mirroring). Seems silly as AFAIK AMD cards, and Quadros don’t have this limitation… In fact the only reason I have a 2nd GPU installed is for running the extra displays, although the extra performance in the few apps that can make use of it is appreciated.

I really want to be done with Windows on my main development ‘battlestation’, I happen to be developing a Linux GPU management program at the moment as well so being able to add my main system as a dev rig would be nice… I don’t even think that a TH2G or a QH2G would really work for me as I’d likely lose things like monitor snapping.

I’ll definitely boot over to Linux in the next it and provide the logs you requested; I guess I’ll reset the Xconfig to the “default” way first, hopefully we can find a way forward. This feels like very basic multi-monitor stuff that should’ve always been supported - I mean, ‘back in the day’ if you wanted multimonitor, you HAD to use two GPUs. The whole thing with every display manager going to a ‘single screen’-type thing I suppose didn’t help keep it easily supported.

If this prime way is only added recently as of 470 - what option was there before that for multi GPU? Just multiple X screens/Xinerama which is pretty much no longer supported anywhere?

I really hope we can get this going; the performance with just one card (the secondary card, 2070S on x1, hooked to my main 3 displays) was droolworthy smooth compared to Windows even. If I can get that kind of performance on all my displays and get most of my main apps setup then I can kick Windows to the curb for the most part.

I suppose this would be even more of a nightmare with different vendor GPUs. I haven’t successfully gotten X to run recognizing both vendor GPUs on my gpu-dev-test rig yet (though that one is technically headless atm); generally the AMD driver crashes with a drm call as if it’s hijacking all the NV drm calls or something.

Help me NV-won-kenobi, you’re my only hope!

The 4-head limit is a hardware limit. Under certain circumstances two or even more connectors can share the same head in mirrored mode, sharing the same PLL.
I’d still need the nvidia-bug-report.log, though.