Runtime D3 support on Lenovo Legion 5 (Ryzen 7 4800H, GTX 1660TI)

Graphics / Linux Linux
1

I have a Legion 5 (AMD Ryzen 7 4800H, GTX 1660TI) and am running Linux. I’m setting up optimus-manager.

[andy@legion5 ~]$ cat /proc/driver/nvidia/gpus/0000:01:00.0/power
Runtime D3 status:          Not supported
Video Memory:               Active

GPU Hardware Support:
 Video Memory Self Refresh: Supported
 Video Memory Off:          Supported

Power Limits:
 Default:                   80000 milliwatts
 GPU Boost:                 4294967295 milliwatts

nvidia-smi output:

[andy@legion5 ~]$ nvidia-smi 
Thu Oct 21 14:10:10 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.74       Driver Version: 470.74       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   38C    P8     5W /  N/A |      5MiB /  5944MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2443      G   /usr/lib/Xorg                       4MiB |
+-----------------------------------------------------------------------------+

Does this mean dynamic power management is not supported on my hardware (and there’s no way to switch it on)?

2

Yeah is not supported.

Did you try to switch to integrated mode in optimus-manager? In my case with a 1050 Ti the NVIDIA powers off and is not consuming battery until I invoke it (in integrated mode as well,no need to switch to hybrid mode anymore).

https://github.com/Askannz/optimus-manager/issues/420#issuecomment-877820312

1 Like
3

Can it be fixed with a BIOS/firmware update (or driver update), or is it pretty much set in concrete? No D3 support for me?

4

http://us.download.nvidia.com/XFree86/Linux-x86_64/470.74/README/dynamicpowermanagement.html

You need to enable a couple stuff for it if you are on pre-Ampere gpu ( which you are )

1 Like
5

Thanks @VulkanGuy! Just to clarify, does this mean it is possible to enable RTD3 on my hardware? I’ll try to follow the document you sent me.

6

Okay, so I’ve managed to get it partially working. This is what I did:

  1. Went in to the BIOS and set the graphics mode to “switchable” (“hybrid”).
  2. Installed the Nvidia driver (version 470.74) from the Arch repositories.
  3. Installed acpi_call (from the Arch repositories) and rebooted.
  4. Installed optimus-manager and optimus-manager-qt (rebooted).

I then configured optimus-manager with the following settings:

    $ cat /etc/optimus-manager/optimus-manager.conf 

    [amd]
    DRI=3
    driver=modesetting
    tearfree=

    [intel]
    DRI=3
    accel=
    driver=modesetting
    modeset=yes
    tearfree=

    [nvidia]
    DPI=96
    PAT=yes
    allow_external_gpus=no
    dynamic_power_management=coarse
    dynamic_power_management_memory_threshold=0
    ignore_abi=yes
    modeset=yes
    options=overclocking, triple_buffer

    [optimus]
    auto_logout=yes
    pci_power_control=no
    pci_remove=yes
    pci_reset=hot_reset
    startup_auto_battery_mode=integrated
    startup_auto_extpower_mode=nvidia
    startup_mode=hybrid
    switching=acpi_call

So basically, in the Optimus tab, set Switching Method to ACPI Call and PCI Reset to Hot Reset. Then in the Nvidia tab, set Dynamic Power Management to Course. I have tried setting Dynamic Power Management to Fine, but then Runtime D3 status shows Not Supported. Not sure if this is a limitation of my card, or if there’s something else I need to do to get “Fine” working.

However, I have noticed that even with Runtime D3 set to Coarse, the Nvidia GPU never powers down completely:

# cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_status
active

# nvidia-smi
Sat Oct 23 16:20:37 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.74       Driver Version: 470.74       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   37C    P8     4W /  N/A |      5MiB /  5944MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1283      G   /usr/lib/Xorg                       4MiB |
+-----------------------------------------------------------------------------+

Shouldn’t the Nvidia GPU power off completely if I’m not using it?

7

@rangelalvarado1 That’s amazing, and exactly where I want to get to!

When I switch to Integrated mode in optimus-manager, I can’t run nvidia-smi or cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_status etc… I take it you still have the Nvidia drivers installed when in Integrated mode?

8

You need to make sure to disable all acpi_calls you made,this disables the nvidia driver:

In my case in optimus-manager I don’t have anything configured,I just let the driver manage by itself

image
image807×587 36.1 KB

This is my optimus-manager.conf

[amd]
DRI=3
driver=modesetting
tearfree=

[intel]
DRI=3
accel=
driver=modesetting
modeset=yes
tearfree=

[nvidia]
DPI=96
PAT=yes
allow_external_gpus=no
dynamic_power_management=no
ignore_abi=no
modeset=yes
options=triple_buffer

[optimus]
auto_logout=yes
pci_power_control=no
pci_remove=no
pci_reset=no
startup_auto_battery_mode=integrated
startup_auto_extpower_mode=hybrid
startup_mode=integrated
switching=none

When you have this,reboot and you should now able to call nvidia-smi in integrated mode.

In case you have TLP,TLP cant prevent the NVIDIA gpu to turn it off properly when you are in integrated mode,you need to do some adjustments to TLP to let the NVIDIA card poweroff.

Basically you need to make sure this settings in /etc/tlp.conf are:

SOUND_POWER_SAVE_ON_AC=1
SOUND_POWER_SAVE_ON_BAT=1

RUNTIME_PM_ON_AC=auto
RUNTIME_PM_ON_BAT=auto

RUNTIME_PM_DENYLIST="01:00.0 01:00.1"

RUNTIME_PM_DENYLIST is the PCI of your NVIDIA card,you can check with lspci,it must be the NVIDIA card and the AUDIO of the card as well,in my case its like this

01:00.0 VGA compatible controller: NVIDIA Corporation GP107M [GeForce GTX 1050 Ti Mobile] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)

After that restart tlp with

sudo tlp start

and now when you do a cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_status it must say suspended

image
image822×299 19.7 KB

9

I tried everything you mentioned, but when I select “Integrated” mode with your Optimus Manager settings, the Nvidia GPU stays in a P3 state, and uses 21W (confirmed by running powertop).

If I start in “Hybrid” mode, the Nvidia GPU stays in a P8 state (using 4W). Again, confirmed using powertop.

In both cases:

$ cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_status
active

Not sure what to look for, but:

$ sudo dmesg | grep -i "D0\|D3\|nvidia\|nouveau"
[    0.000000] Command line: initrd=\amd-ucode.img initrd=\initramfs-linux.img cryptdevice=PARTUUID=23853d28-971e-46db-bb2d-dd3ae0afd208:luksdev root=/dev/mapper/luksdev rw intel_pstate=no_hwp nvidia-drm.modeset=1
[    0.000000] BIOS-e820: [mem 0x0000000009f0d000-0x00000000bab68fff] usable
[    0.000000] reserve setup_data: [mem 0x0000000009f0d000-0x00000000b50fc017] usable
[    0.000000] efi: ACPI=0xcdffe000 ACPI 2.0=0xcdffe014 TPMFinalLog=0xcdedc000 SMBIOS=0xcb70d000 SMBIOS 3.0=0xcb70b000 ESRT=0xb64acb98 MEMATTR=0xb512e018 RNG=0xcb708b18 TPMEventLog=0xb512a018 
[    0.005287] ACPI: BOOT 0x00000000CDFED000 000028 (v01 LENOVO CB-01    00000001      01000013)
[    0.005308] ACPI: CRAT 0x00000000CDFFD000 000F28 (v01 LENOVO CB-01    00000001 AMD  00000001)
[    0.005314] ACPI: SSDT 0x00000000CDFBD000 000D37 (v01 LENOVO CB-01    00000001 INTL 20130117)
[    0.005337] ACPI: Reserving BOOT table memory at [mem 0xcdfed000-0xcdfed027]
[    0.005344] ACPI: Reserving CRAT table memory at [mem 0xcdffd000-0xcdffdf27]
[    0.005346] ACPI: Reserving SSDT table memory at [mem 0xcdfbd000-0xcdfbdd36]
[    0.005452]   node   0: [mem 0x0000000009f0d000-0x00000000bab68fff]
[    0.057787] ACPI: HPET id: 0x10228210 base: 0xfed00000
[    0.061813] Kernel command line: initrd=\amd-ucode.img initrd=\initramfs-linux.img cryptdevice=PARTUUID=23853d28-971e-46db-bb2d-dd3ae0afd208:luksdev root=/dev/mapper/luksdev rw intel_pstate=no_hwp nvidia-drm.modeset=1
[    0.061867] Unknown command line parameters: cryptdevice=PARTUUID=23853d28-971e-46db-bb2d-dd3ae0afd208:luksdev
[    0.343081] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.343087] pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff window]
[    0.343093] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xf7ffffff window]
[    0.343093] pci_bus 0000:00: root bus resource [mem 0xfc000000-0xfed3ffff window]
[    0.343412] pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
[    0.343565] pci 0000:00:01.2: PME# supported from D0 D3hot D3cold
[    0.343746] pci 0000:00:02.1: PME# supported from D0 D3hot D3cold
[    0.343872] pci 0000:00:02.2: PME# supported from D0 D3hot D3cold
[    0.344003] pci 0000:00:02.4: PME# supported from D0 D3hot D3cold
[    0.344167] pci 0000:00:08.1: PME# supported from D0 D3hot D3cold
[    0.344275] pci 0000:00:08.2: PME# supported from D0 D3hot D3cold
[    0.344857] pci 0000:01:00.0: reg 0x10: [mem 0xd0000000-0xd0ffffff]
[    0.344874] pci 0000:01:00.0: reg 0x1c: [mem 0x1ffd0000000-0x1ffd1ffffff 64bit pref]
[    0.344935] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    0.345457] pci 0000:01:00.2: PME# supported from D0 D3hot D3cold
[    0.345581] pci 0000:01:00.3: PME# supported from D0 D3hot D3cold
[    0.345639] pci 0000:00:01.1:   bridge window [mem 0xd0000000-0xd10fffff]
[    0.346281] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.346816] pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
[    0.347593] pci 0000:06:00.0: PME# supported from D1 D2 D3hot D3cold
[    0.347821] pci 0000:06:00.3: PME# supported from D0 D3hot D3cold
[    0.347926] pci 0000:06:00.4: PME# supported from D0 D3hot D3cold
[    0.348030] pci 0000:06:00.5: PME# supported from D0 D3hot D3cold
[    0.348130] pci 0000:06:00.6: PME# supported from D0 D3hot D3cold
[    0.357546] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.362232] system 00:03: [io  0x04d0-0x04d1] has been reserved
[    0.362238] system 00:03: [io  0x0cd0-0x0cdb] has been reserved
[    0.376412] pci 0000:00:01.1:   bridge window [mem 0xd0000000-0xd10fffff]
[    0.376472] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.376478] pci_bus 0000:00: resource 11 [mem 0x000d0000-0x000d3fff window]
[    0.376484] pci_bus 0000:00: resource 19 [mem 0xd0000000-0xf7ffffff window]
[    0.376485] pci_bus 0000:00: resource 20 [mem 0xfc000000-0xfed3ffff window]
[    0.376488] pci_bus 0000:01: resource 1 [mem 0xd0000000-0xd10fffff]
[    0.376688] pci 0000:01:00.1: D0 power state depends on 0000:01:00.0
[    0.376758] pci 0000:01:00.2: D0 power state depends on 0000:01:00.0
[    0.376990] pci 0000:01:00.3: D0 power state depends on 0000:01:00.0
[    0.377464] pci 0000:06:00.3: extending delay after power-on from D3hot to 20 msec
[    0.404626] pci 0000:06:00.4: extending delay after power-on from D3hot to 20 msec
[    0.475720] ACPI: button: Lid Switch [LID0]
[    0.830473]     cryptdevice=PARTUUID=23853d28-971e-46db-bb2d-dd3ae0afd208:luksdev
[    1.322364] usb 3-3: New USB device found, idVendor=13d3, idProduct=56ff, bcdDevice=19.19
[   17.907208] nvidia-gpu 0000:01:00.3: enabling device (0000 -> 0002)
[   18.191881] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input12
[   18.191963] snd_hda_codec_realtek hdaudioC1D0: autoconfig for ALC257: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:speaker
[   18.191967] snd_hda_codec_realtek hdaudioC1D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[   18.191968] snd_hda_codec_realtek hdaudioC1D0:    hp_outs=1 (0x21/0x0/0x0/0x0/0x0)
[   18.191969] snd_hda_codec_realtek hdaudioC1D0:    mono: mono_out=0x0
[   18.191970] snd_hda_codec_realtek hdaudioC1D0:    inputs:
[   18.191971] snd_hda_codec_realtek hdaudioC1D0:      Mic=0x19
[   18.191972] snd_hda_codec_realtek hdaudioC1D0:      Internal Mic=0x12
[   18.205164] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input13
[   18.230689] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input14
[   18.230762] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input15
[   18.230814] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:01.1/0000:01:00.1/sound/card0/input16
[   19.183785] nvidia-gpu 0000:01:00.3: i2c timeout error e0000000
[   19.368084] usb 3-3: Found UVC 1.00 device Integrated Camera (13d3:56ff)
[   19.865808] nvidia: loading out-of-tree module taints kernel.
[   19.865819] nvidia: module license 'NVIDIA' taints kernel.
[   19.893844] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[   19.945989] nvidia-nvlink: Nvlink Core is being initialized, major device number 508
[   19.946478] nvidia 0000:01:00.0: enabling device (0000 -> 0003)
[   19.946584] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[   19.988982] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  470.74  Mon Sep 13 23:09:15 UTC 2021
[   20.023366] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  470.74  Mon Sep 13 22:59:50 UTC 2021
[   20.027526] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[   20.813276] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1

Really stumped on this…

10

The only thing I can think off is maybe you don’t have the NVIDIA tweaks that NVIDIA recommends in Automated Setup

For example in Manjaro I have this file

/etc/udev/rules.d/90-mhwd-prime-powermanagement.rules

and contains

# Remove NVIDIA USB xHCI Host Controller devices, if present
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x0c0330", ATTR{remove}="1"

# Remove NVIDIA USB Type-C UCSI devices, if present
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x0c8000", ATTR{remove}="1"

# Remove NVIDIA Audio devices, if present
#ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x040300", ATTR{remove}="1"

# Enable runtime PM for NVIDIA VGA/3D controller devices on driver bind
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="auto"
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="auto"

# Disable runtime PM for NVIDIA VGA/3D controller devices on driver unbind
ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"
ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"

I think if you don’t have the Enabled runtime PM the NVIDIA card will not manage power management by itself.
Or unless the GTX 1660 Ti behave different than a GTX 1050 Ti.

11

I have those udev rules set. At first I skipped the first two (as per the instruction in Chapter 22), but I also tried with all of those udev rules.

12

Very weird to be honest,maybe is something Manjaro is doing? or it something with KDE power management,I don’t know. :/

13

I own a laptop with a Ryzen 4800H and RTX 2060 and I’m also in the same situation as you. The (fine-grained) mode doesn’t work and shows Runtime D3 status: Not Supported. The (coarse) mode works just fine, like yours.

It seems that maybe we out of luck, because I saw this comment and the guy in that thread have a Ryzen 5900H and /proc/driver/nvidia/gpus/0000:01:00.0/power shows Runtime D3 status: Enabled (fine-grained).

I decompiled my ACPI tables and shows the _PR0 and _PR3 methods but still it doesn’t work with my 4000s series Ryzen. It could be that maybe there’s something in the latests Ryzen 5000s series that add support to RTD3 or maybe the nvidia driver excludes most of the Ryzen mobile APUs. But that is something that a Nvidia developer must clarified

1 Like
14

Thanks for the info! Do you think there’s a possibility that something in /usr/lib/Xorg is keeping the Nvidia GPU powered on?

RTD3 is such a cool feature, would be great to get it working in Linux on AMD 4000 series laptops. Would be interesting to know if it works on our laptops in Windows (I just have no idea how to check something like that in Windows).

If all else fails, I might just purchase a new laptop with an RTX 3000 series GPU. From what I’ve read, RTD3 is enabled by default on the 3000 series Nvidia GPUs.

15

@AndyTurfer I don’t think that Xorg is keeping the nvidia powered on, the think here is that the (fine-grained) mode cannot be enabled in AMD Ryzen 4000s series, that’s why the nvidia GPU keeps awake. This is very likely a driver limitation (hopefully) and not a hardware limitation.

In windows the nvidia GPU power off when you are not using it. You can check it with HWiNFO, all the clocks and power consumption goes to 0 after a couple of seconds of inactivity in the GPU, that’s why my laptop and much others have a better battery life while in Windows.

So, technically the GPU CAN be turned off, it just doesn’t work under linux, that’s why I think that this is just an issue on the driver, but WE need confirmation of nvidia developers.

1 Like
16

BTW - I can get better battery life in Linux, but I have to completely power down the Nvidia GPU (using optimus-manager).

17

Yeah, me too, but only with the Nvidia GPU turned off (like you). Right now I use acpi_call with \_SB.PCI0.GPP0.PG00._OFF and \_SB.PCI0.GPP0.PG00._ON to turn it off and on. But I will be better if RTD3 worked as expected in this laptops.

2 Likes
18

EDIT: Yeah that process (at least on pre-turing cards) keeps the NVIDIA card powered on,the card only power off when there is no process in the GPU.

It can powered off at least for me,but the thing is,I don’t know why its working for me with a pascal card when supposedly is not supported on Linux unless you have the Runtime D3 support.


I really want to know why its working because I was helping another guy and it can’t turn off their NVIDIA card too.

2 Likes
19

@rangelalvarado1 wow, really nice. Supposedly all of your hardware is not “supported” by RTD3 and still it works as expected. What does cat /proc/driver/nvidia/gpus/0000\:01\:00.0/power shows

20

A bunch of ?

File: /proc/driver/nvidia/gpus/0000:01:00.0/power
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ Runtime D3 status:          ?
   2   │ Video Memory:               ?
   3   │ 
   4   │ GPU Hardware Support:
   5   │  Video Memory Self Refresh: ?
   6   │  Video Memory Off:          ?
   7   │ 
   8   │ Power Limits:
   9   │  Default:                   N/A milliwatts
  10   │  GPU Boost:                 N/A milliwatts