How to force graphics to run on specific GPU with nvidia-prime

Graphics / Linux Linux
1

My computer has one integrated graphics card and 2 Nvidia RTX 3070 GPUS. I am using Ubuntu 20.04 and nvidia-driver-530.

lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation AlderLake-S GT1 (rev 0c)
01:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Lite Hash Rate] (rev a1)
05:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Lite Hash Rate] (rev a1)

I am currently trying to test my 3070 graphics cards with the Phoronix Test Suite.

I am using nvidia-prime and prime-select: on-demand to run the terminal on the intel iGPU and phoronix tests on the Nvidia 3070: prime-run phoronix-test-suite run unigine-heaven.

There were some issues getting nvidia-prime to work, so I followed the suggestions from this article:

  • askubuntu 1364762
cat /usr/bin/prime-run
#!/bin/bash
export __NV_PRIME_RENDER_OFFLOAD=1
export __GLX_VENDOR_LIBRARY_NAME=nvidia
export __VK_LAYER_NV_optimus=NVIDIA_only
export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json
exec "$@"

By using prime-run I am successfully able to run the phoronix test suite on GPU 0 which has bus id 01:00.0 / PCI:1:0:0.

However, I seem unable to run any tests with GPU 1 which has bus id 05:00.0 / PCI:5:0:0.

Modifying /etc/X11/xorg.conf by changing the bus number and rebooting as suggested by the following links didn’t seem to do anything and still ran on GPU 0.

  • stackoverflow 18382271
  • askubuntu 787030
cat /etc/X11/xorg.conf
# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig:  version 530.41.03

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0"
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
EndSection

Section "Files"
EndSection

Section "InputDevice"
    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"
    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Unknown"
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "5:0:0"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

I deleted etc/X11/xorg.conf and was able to run the phoronix tests on GPU 0 without the conf file at all. I would guess that one of the drivers or programs I run automatically selects the nvidia card with the lowest bus id.

I would like to know where I should look to change the settings or any configuration files in order to select the second RTX 3070 gpu with the bus id 05:00.0 for running the phoronix test suite. I would be more than happy to provide any further information.

nvidia-bug-report.log.gz (1.2 MB)

2

Also set
__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0
for the first gpu or
__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G1
for the second.
https://download.nvidia.com/XFree86/Linux-x86_64/495.44/README/primerenderoffload.html
This doesn’t work for vulkan, though. This would need something like
https://github.com/aejsmith/vkdevicechooser

3

That looks like it should work in theory. However, I am unable to get NVIDIA-G0 or NVIDIA-G1 from xrandr --listproviders .

xrandr --listproviders
Providers: number : 0

Because of this, it seems I am unable to select the __NV_PRIME_RENDER_OFFLOAD_PROVIDER to be either of the two cards.

Would you know how to resolve this?

4

You’re running a wayland session.

5

I think I did not mention before, but I am accessing the server running Ubuntu server through ssh -X from a standard Ubuntu desktop computer.

loginctl
SESSION  UID USER   SEAT TTY  
      1 1000 tianzi      pts/0

1 sessions listed.

loginctl show-session 1 -p Type
Type=tty

I have confirmed that my server is running Xwayland

xrandr --listmonitors
Monitors: 1
 0: +*XWAYLAND15 3840/620x2160/330+0+0  XWAYLAND15

I have commented out WaylandEnable=false in /etc/gdm3/custom.conf, but I don’t think that changed xrandr --listproviders.

cat /etc/gdm3/custom.conf 
# GDM configuration storage
#
# See /usr/share/gdm/gdm.schemas for a list of available options.

[daemon]
# Uncomment the line below to force the login screen to use Xorg
WaylandEnable=false

# Enabling automatic login
#  AutomaticLoginEnable = true
#  AutomaticLogin = user1

# Enabling timed login
#  TimedLoginEnable = true
#  TimedLogin = user1
#  TimedLoginDelay = 10

[security]

[xdmcp]

[chooser]

[debug]
# Uncomment the line below to turn on debugging
# More verbose logs
# Additionally lets the X server dump core if it crashes
#Enable=true
6

I have no idea what you’re trying to, especially when using X forwarding. This means rendering is happening on your local Xserver.

7

Yes, I have a server and a desktop computer. My server has the 2 Nvidia Graphics cards, and I want to run the phoronix test suite on those two server graphics cards.

I am accessing the server from my desktop computer with ssh -X. However when I try to get the graphics cards on the server recognized with xrandr --listproviders, I am unable to get anything. I believe this is because the server is running wayland.

After trying to disable wayland on the server in /etc/gdm3/custom.conf, I still run into the same issues, and it appears the server is connecting to my desktop computer’s monitor using Xwayland. So it appears my attempt to disable wayland was unsuccessful.

My thought is that disabling wayland entirely should be able to get the server to use X11 to provide graphics and have the server recognize the two Nvidia graphics cards for xrandr --listproviders.

8

You have a complete misunderstanding about how X forwarding works and what it does. Please read about the DISPLAY variable and how to connect over ssh to a remote Xserver running on the server.

9

The $DISPLAY variable when run from my desktop machine’s terminal yields

echo $DISPLAY
:0

When run on the virtual terminal over ssh to the server, I get

echo $DISPLAY
localhost:10.0

I think the problem is that the X forwarding is done on top of a Xwayland compatibility layer.
The remote server display works well on the phoronix test suite when it automatically selects the first Nvidia graphics card.

The problem is that I cannot select which Nvidia graphics card provider I want to at will.
This is because xrandr --listproviders doesn’t return me anything.

I want the rendering to happen on the Xserver. For some of the phoronix test suites, it doesn’t really matter that the graphics is piped to my computer’s desktop monitor or not since I only want to stress test the cards.

Right now, all I want is the capability to select which Nvidia Graphics cards that I am using to do the test.

I think I have realized that gdm3 doesn’t really have anything to do with my issue since it appears unrelated, so I have purged it from the system.

Does that clarify the issue that I am having?
How would I get it so that xrandr --listproviders actually provides me the graphics cards on the system?

Thanks!

10

Ultimately, what I was trying to do did not seem possible.

Since I do not think the graphics software is meant to be run this way on a headless Ubuntu server, I installed lightdm so I could access a GUI on the server machine and connect it directly to the monitor with a direct video connection.

With the GUI installed, everything seemed to work fine with all the Nvidia cards being recognized.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.