Is it possible to get prime to work on two dGPUs instead of iGPU + dGPU? [Wayland]

Graphics / Linux Linux
1

I currently have two GPUs connected to my PC: a GTX 1060 that drives all my monitors, and an RTX 3070 that currently doesn’t have anything hooked up to it. The 3070’s driver is blocked from loading during boot via vfio-pci, and I am manually re-binding it to the nvidia driver after boot. After this, nvidia-smi reports both cards as active.

I’ve been trying to use Nvidia PRIME with prime-run to get my games to run on the “headless” 3070, and send the rendered frames back to the 1060, similarly to how it would work on any modern gaming laptop with an iGPU+dGPU setup. Unfortunately, I haven’t been able to get anything to actually use the 3070 through prime, even though both cards are connected and working under the nvidia driver.

I even tried selecting the card by ID using __NV_PRIME_RENDER_OFFLOAD_PROVIDER, but it doesn’t make a difference. here’s an example glxgears output:

__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G1 prime-run glxgears -info | grep -i -e "GL_RENDERER" -e "GL_VERSION"
GL_RENDERER   = NVIDIA GeForce GTX 1060 6GB/PCIe/SSE2
GL_VERSION    = 4.6.0 NVIDIA 555.52.04

According to nvidia-smi, the ID of the second card is 1, so the command above should be forcing glxgears to use the 3070, but it still only uses the 1060.

The reason I want to do all this is because occasionally I want to be able to use the 3070 in a Windows 11 VM via VFIO passthrough, without having to shut down my host Wayland session.

I’ve seen people succeed with this, even on the desktop, by connecting a single monitor to their CPU’s built-in Intel GPU, but I can’t do that because I have more monitors than there are HDMI/DP ports on my motherboard. Because of this, I have to use two dGPUs.

So I guess my question is, can prime render offload work between two Nvidia GPUs instead of an Intel iGPU and a dedicated Nvidia dGPU? And if so, what am I doing wrong with the commands above, and why aren’t my graphics apps using the 3070 when run through prime-run?

Here are some extra command outputs:

nvidia-smi

lspci -nnk -d 10de:1c03; lspci -nnk -d 10de:2484

2

It “should” work when using Xorg. The last time a user checked, it didn’t but that might have been fixed meanwhile.
You’re running a wayland session so there actually are no “providers” to choose from. IDK if (X)wayland has a method to specify a gpu.
Using Vulkan, vkdevicechooser might be used to select the gpu.

3

I have two T1000 working with nvidia2nvidia PRIME and X11. The problem is with the drivers. I have to use very specific sub version of 520 driver (don’t remember which, not the newest one). Not working good with 515 (problem with offloading). Not working at all with 525 (totaly hang). I have one try with 535 and also no no. Give a try with the driver 520.

5

I have two T1000 working with nvidia2nvidia PRIME and X11. The problem is with the drivers. I have to use very specific sub version of 520 driver (don’t remember which, not the newest one). Not working good with 515 (problem with offloading). Not working at all with 525 (totaly hang). I have one try with 535 and also no no. Give a try with the driver 520.

The problem is that I need to use the 555 beta driver, because it fixes some critical issues on Wayland.

6

Most popular distros will do the PRIME magic out of the box but obviously not if have 3xxx blocked at the start of the system. Show us xrandr --listproviders at the time when PRIME should working in your opinion.

7

Most popular distros will do the PRIME magic out of the box but obviously not if have 3xxx blocked at the start of the system.

It is blocked at boot, but I am manually loading the nvidia driver on it after I’ve logged in. And that part seems to work, because nvidia-smi detects both cards when the nvidia driver is loaded on both of them.

Show us xrandr --listproviders at the time when PRIME should working in your opinion.

I’m on Wayland, so that command returns nothing (something like “0 providers” if I remember correctly).