Nvidia-settings: Attribute GPUMemoryTransferRateOffsetAllPerformanceLevels (...) is not available

Graphics / Linux Linux
1

When running

nvidia-settings --verbose -a "GPUMemoryTransferRateOffsetAllPerformanceLevels=1270"

the command has no effect and I’m getting the following warning:

WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:0].

See attached files for system configuration and log files.
Thank you in advance for your help.

Output from nvidia-settings:

WARNING: NV-CONTROL extension not present on screen 0 of this Display.
WARNING: Unable to determine number of NVIDIA Frame Lock Devices on ':0'.
WARNING: Unable to determine number of NVIDIA 3D Vision Pro Transceivers on ':0'.
WARNING: Unable to determine number of NVIDIA Display Devices on ':0'.
WARNING: Unable to determine number of NVIDIA X Screens on ':0'.
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:0].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:1].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:2].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:3].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:4].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:5].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:6].

Output from nvidia-smi:

Tue Jan 26 00:22:11 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   24C    P8     5W / 151W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1070    Off  | 00000000:02:00.0 Off |                  N/A |
|  0%   28C    P8     7W / 180W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 1070    Off  | 00000000:03:00.0 Off |                  N/A |
|  0%   29C    P8     7W / 180W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 1070    Off  | 00000000:06:00.0 Off |                  N/A |
|  0%   27C    P8     4W / 151W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 1070    Off  | 00000000:07:00.0 Off |                  N/A |
|  0%   20C    P8     4W / 151W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 1070    Off  | 00000000:09:00.0 Off |                  N/A |
|  0%   31C    P8     7W / 180W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX 1070    Off  | 00000000:0A:00.0 Off |                  N/A |
|  0%   29C    P8     4W / 151W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1042      G   /usr/libexec/Xorg                   4MiB |
|    1   N/A  N/A      1042      G   /usr/libexec/Xorg                   4MiB |
|    2   N/A  N/A      1042      G   /usr/libexec/Xorg                   4MiB |
|    3   N/A  N/A      1042      G   /usr/libexec/Xorg                   4MiB |
|    4   N/A  N/A      1042      G   /usr/libexec/Xorg                   4MiB |
|    5   N/A  N/A      1042      G   /usr/libexec/Xorg                   4MiB |
|    6   N/A  N/A      1042      G   /usr/libexec/Xorg                   4MiB |
+-----------------------------------------------------------------------------+

Output of lspci:

00:00.0 Host bridge: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 08)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 08)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630
00:14.0 USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]
00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)
00:1b.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0)
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
00:1c.3 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #4 (rev f0)
00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0)
00:1c.5 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #6 (rev f0)
00:1c.6 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #7 (rev f0)
00:1c.7 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #8 (rev f0)
00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Z370 Chipset LPC/eSPI Controller
00:1f.2 Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
00:1f.4 SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
03:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
03:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
06:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
06:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
07:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
09:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
09:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
0a:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
0a:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

Installed packages:

Installed Packages
akmod-nvidia.x86_64                       3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
kmod-nvidia-5.10.7-200.fc33.x86_64.x86_64 3:460.32.03-1.fc33 @@commandline             
kmod-nvidia-5.10.9-201.fc33.x86_64.x86_64 3:460.32.03-1.fc33 @@commandline             
kmod-nvidia-5.9.16-200.fc33.x86_64.x86_64 3:460.32.03-1.fc33 @@commandline             
nvidia-persistenced.x86_64                3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
nvidia-settings.x86_64                    3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
nvidia-xconfig.x86_64                     3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia.x86_64                3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda.x86_64           3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda-libs.x86_64      3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-devel.x86_64          3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-kmodsrc.x86_64        3:460.32.03-1.fc33 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-libs.x86_64           3:460.32.03-1.fc33 @rpmfusion-nonfree-updates

boot.log (218.1 KB)
xorg.conf (2.0 KB)
Xorg.0.log (175.7 KB)

2

I guess you’re trying to control the screen that’s driven by the intel gpu (:0.0) while the nvidia gpus are on :0.1…7

3

Ok.
How can I verify this?
How to modify failing command to take this into account?

4

–display=:0.1

nvidia-settings -h

5

It is strange but --display is not listed as an option neither in the output of nvidia-settings --help nor in the man nvidia-settings. It’s only mentioned in the description of other options like --ctrl-display.
I raised Option --display absent from help and man page · Issue #60 · NVIDIA/nvidia-settings · GitHub

I guess you’re trying to control the screen that’s driven by the intel gpu (:0.0) while the nvidia gpus are on :0.1…7

How can I see what screen does each gpu use?
I tried the following:

$ nvidia-settings --display=:0 --query=screens --verbose

WARNING: NV-CONTROL extension not present on screen 0 of this Display.
WARNING: Unable to determine number of NVIDIA Frame Lock Devices on ':0'.
WARNING: Unable to determine number of NVIDIA 3D Vision Pro Transceivers on ':0'.
WARNING: Unable to determine number of NVIDIA Display Devices on ':0'.
WARNING: Unable to determine number of NVIDIA X Screens on ':0'.
WARNING: No X Screens on hostx:0

Changing --display=:0 to --ctrl-display=:0 gives the same output.

Please notice Is not driving any X Screen below:

$ nvidia-settings --ctrl-display=:0 --query=gpus --verbose

WARNING: NV-CONTROL extension not present on screen 0 of this Display.
WARNING: Unable to determine number of NVIDIA Frame Lock Devices on ':0'.
WARNING: Unable to determine number of NVIDIA 3D Vision Pro Transceivers on ':0'.
WARNING: Unable to determine number of NVIDIA Display Devices on ':0'.
WARNING: Unable to determine number of NVIDIA X Screens on ':0'.

7 GPUs on hostx:0

    [0] hostx:0[gpu:0] (GeForce GTX 1070)
      Has the following names:
        GPU-0
        GPU-2e139785-c590-8c78-3ce3-84754181c89e
      Is not driving any X Screen.
      Does not support any Display Device.
      Is not connected to any Frame Lock Device.
      Is connected to the following Fan:
        hostx:0[fan:0] (Fan 0)
      Is connected to the following Thermal Sensor:
        hostx:0[thermalsensor:0] (Thermal Sensor 0)

    [1] hostx:0[gpu:1] (GeForce GTX 1070)
      Has the following names:
        GPU-1
        GPU-a7fc007b-9e46-6e6e-df14-9aada5422fa7
      Is not driving any X Screen.
      Does not support any Display Device.
      Is not connected to any Frame Lock Device.
      Is connected to the following Fan:
        hostx:0[fan:1] (Fan 1)
      Is connected to the following Thermal Sensor:
        hostx:0[thermalsensor:1] (Thermal Sensor 1)

    [2] hostx:0[gpu:2] (GeForce GTX 1070)
      Has the following names:
        GPU-2
        GPU-cb4d9c5b-d61a-1c98-eab3-5ab35246f533
      Is not driving any X Screen.
      Does not support any Display Device.
      Is not connected to any Frame Lock Device.
      Is connected to the following Fan:
        hostx:0[fan:2] (Fan 2)
      Is connected to the following Thermal Sensor:
        hostx:0[thermalsensor:2] (Thermal Sensor 2)

    [3] hostx:0[gpu:3] (GeForce GTX 1070)

      Has the following names:
        GPU-3
        GPU-f876750b-4074-e19e-4607-7d05e46be5dd
      Is not driving any X Screen.
      Does not support any Display Device.
      Is not connected to any Frame Lock Device.
      Is connected to the following Fan:
        hostx:0[fan:3] (Fan 3)
      Is connected to the following Thermal Sensor:
        hostx:0[thermalsensor:3] (Thermal Sensor 3)

    [4] hostx:0[gpu:4] (GeForce GTX 1070)
      Has the following names:
        GPU-4
        GPU-e798ae46-696c-c47a-2f84-fecd270fa689
      Is not driving any X Screen.
      Does not support any Display Device.
      Is not connected to any Frame Lock Device.
      Is connected to the following Fan:
        hostx:0[fan:4] (Fan 4)
      Is connected to the following Thermal Sensor:
        hostx:0[thermalsensor:4] (Thermal Sensor 4)

    [5] hostx:0[gpu:5] (GeForce GTX 1070)

      Has the following names:
        GPU-5
        GPU-6f4871f6-daa9-f926-0799-301679ff3627
      Is not driving any X Screen.
      Does not support any Display Device.
      Is not connected to any Frame Lock Device.
      Is connected to the following Fan:
        hostx:0[fan:5] (Fan 5)
      Is connected to the following Thermal Sensor:
        hostx:0[thermalsensor:5] (Thermal Sensor 5)

    [6] hostx:0[gpu:6] (GeForce GTX 1070)
      Has the following names:
        GPU-6
        GPU-49c7ea97-cd24-2ed5-9b1a-ccb2ed184f68
      Is not driving any X Screen.
      Does not support any Display Device.
      Is not connected to any Frame Lock Device.
      Is connected to the following Fan:
        hostx:0[fan:6] (Fan 6)
      Is connected to the following Thermal Sensor:
        hostx:0[thermalsensor:6] (Thermal Sensor 6)

The odd thing is that moving the following sections for Intel’s builit-in graphics in xorg.conf after sections for Nvidia’s gpus:

Section "Device"
    Identifier "intel"
    Driver "modesetting"
    BusID "PCI:0:2:0"
EndSection

Section "Screen"
    Identifier "intel"
    Device "intel"
EndSection

changes behaviour of nvidia-settings.
Now it is possible to control one of Nvidia’s gpus:

$ nvidia-settings --verbose --display=:0 -a GPUMemoryTransferRateOffsetAllPerformanceLevels=1270

libEGL warning: DRI2: failed to authenticate

WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:1].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:2].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:3].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:4].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:5].
WARNING: Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' specified in assignment 'GPUMemoryTransferRateOffsetAllPerformanceLevels=1270' is not available on hostx:0[gpu:6].

Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' (hostx:0.0) assigned value 1270.
Attribute 'GPUMemoryTransferRateOffsetAllPerformanceLevels' (hostx:0[gpu:0]) assigned value 1270.

We see that this one gpu is now driving X screen:

$ nvidia-settings --verbose --display=:0 --query=gpus

libEGL warning: DRI2: failed to authenticate

7 GPUs on hostx:0

    [0] hostx:0[gpu:0] (GeForce GTX 1070)

      Has the following names:
        GPU-0
        GPU-2e139785-c590-8c78-3ce3-84754181c89e

      Is driving the following X Screen:
        hostx:0.0 (X Screen 0)
(...)

…and that we have one X screen to drive:

$ nvidia-settings --verbose --display=:0 --query=screens

libEGL warning: DRI2: failed to authenticate

1 X Screen on hostx:0

    [0] hostx:0.0 (GeForce GTX 1070)

      Has the following name:
        SCREEN-0

      Is driven by the following GPU:
        hostx:0[gpu:0] (GeForce GTX 1070)

      Is assigned the following Display Devices:
        hostx:0[dpy:0] (DVI-D-0)
        hostx:0[dpy:1] (HDMI-0)
        hostx:0[dpy:2] (DP-0)
        hostx:0[dpy:3] (DP-1)
        hostx:0[dpy:4] (DP-2)
        hostx:0[dpy:5] (DP-3)
        hostx:0[dpy:6] (DP-4)
        hostx:0[dpy:7] (DP-5)

The questions are:

Is it normal that moving Device and Screen sections around in xorg.conf changes behaviour of X and subsequently of nvidia-settings? As there is ServerLayout section listing all screens one would assume this should mandate order of screens and the order in which Screen sections appear in the file should not matter.
It is hard to interpret what does the following from man page of xorg.conf really mean to the order of screens:

A Screen section is considered “active” if it is referenced by an active ServerLayout section or by the −screen command line option. If neither of those is present, the first Screen section found in the config file is considered the active one.

Why is only one of the screens using Nvidia’s gpus active?

I really need some help to make sense out of all of this.

6

I’m getting curious here…
I don’t seem to have the attribute GPUMemoryTransferRateOffsetAllPerformanceLevels in my nvidia-settings.
So it seems I cannot lookup if it’s appropriate with the gpu target.
Would you mind to try setting the attribute for all the gpus like this?:

for ((x=0;x<7;x++)); do
  nvidia-settings --verbose --display=:0 -a "[gpu:${x}]/GPUMemoryTransferRateOffsetAllPerformanceLevels=1270"
done

Taking the numbers from your nvidia-smi output. if that’s not correct you might need to change the numbers “0” and “7” in the arithmetic expression.

Edit: maybe you can skip the --display parameter…

7

Hi Mart!

Thanks for joining in and trying to help.
I run your snippet and the result was the same as I originally reported – warning about attribute not being available for every nvidia gpu with my original xorg.conf and the command working for one nvidia gpu and failing for all other nvidia gpus after moving sections in xorg.conf as described earlier.

8

Quite a brain twister that issue :-O

So the problems seems your screen definitions actually not being recognized (except one).

I’ve spent some time now reading the xorg.conf man page (not so familiar with it, as I never needed it).
From what I can tell, your config should be good.
Maybe it’s a parsing issue. Just a shot in the dark:
Try to put the ServerLayout section at the end of the config.
The ServerLayout section should be active, as per documentation, but maybe worth a shot adding a ServerFlags section with the DefaultServerLayout option.
Also maybe there’s a problem with no Monitor section. From Screen description:

Monitor “monitor−id”

specifies which monitor description is to be used for this screen. If a Monitor name is not specified, a default configuration is used. Currently the default configuration may not function as expected on all platforms.

But just another shot in the dark… Because what exactly does “may not function as expected on all platforms” mean… and you specified Option “UseDisplayDevice” “none” for each screen.

Still also I’d be curious if the behavior of the nvidia-settings command changes when specifying a screen id:

for ((x=0;x<7;x++)); do
  nvidia-settings --verbose -a :0.$((x+1))[gpu:${x}]/GPUMemoryTransferRateOffsetAllPerformanceLevels=1270
done

Edit: maybe also try omitting the [gpu:${x}] part.

I’m sorry if I can’t come up with “This is the solution…”, just with maybe useless things to try.
Still I’d create a bug report by running nvidia-bug-report.sh as root and attaching the created file here.
Maybe that will give a little insight. Also it’s quite a requirement for getting help here.
Maybe some Nvidia employee can join in and give some expert insight? :)

9

It turned out the problem was due to the /usr/share/X11/xorg.conf.d/nvidia.conf file which comes with the xorg-x11-drv-nvidia package (version 460.39 in this case) which looks like this:

#This file is provided by xorg-x11-drv-nvidia
#Do not edit

Section "OutputClass"
	Identifier "nvidia"
	MatchDriver "nvidia-drm"
	Driver "nvidia"
	Option "AllowEmptyInitialConfiguration"
	Option "SLI" "Auto"
	Option "BaseMosaic" "on"
EndSection

Section "ServerLayout"
	Identifier "layout"
	Option "AllowNVIDIAGPUScreens"
EndSection

where there is ServerLayout section which apparently has overridden my section with the same name from /etc/X11/xorg.conf file. Removing this file fixed the problem.

I’m wondering if there is a way to see the final configuration constructed by X server after having read configuration files from all possible locations. This way one could diagnose such problems easily.