Enabling PCIe 3.0 with NVreg_EnablePCIeGen3 on Titan

Graphics / Linux Linux
1

Hi, I’m running two Titans on Ubuntu 12.04 64-bit (kernel version 3.5.0-31) with the Nvidia 319.23 driver and I’m trying to enable PCIe 3.0 using the line

NVreg_EnablePCIeGen3=1

in my .nvidia-settings-rc file (is this the right place?), but it doesn’t seem to work. At least, when I query PCI under full load with

lspci -vv -i 03:00.0 | grep LnkSta

as suggested elsewhere, all of my x16 slots are reporting 5GT/s instead of 8GT/s. PCIe 3.0 is enabled in the BIOS.

Not seeing any hints… any help?

Another possibly related question: I’m using my (non-nvidia) onboard display adapter for my monitor, so when the nvidia driver offered to update my xorg.conf file, I declined. Am I missing something and need to run this for some reason?

As you can see from my confusion, any help would be very useful. Thanks!

2

y think is in your modprobe conf

or when load module:

modprobe nvidia NVreg_EnablePCIeGen3=1

greetings

3

Edit /etc/modprobe.d/local.conf or create a new file like /etc/modprobe.d/nvidia.conf

and add this:

options nvidia NVreg_EnablePCIeGen3=1
4

I’ve added this option, updated grub-efi, updated initramfs, but strange thing is that after reboot it works only if I start X once, then exit X, unload the module, and load it again.
Just reloading the module after system boot before X has no effect. What could be wrong?

5

Does your initramfs include and load nvidia.ko on boot? If so you have to change initramfs modutils settings - I’ve no idea how to do that - that’s distro specific.

OK, here’s a different solution which should work no matter what.

Update your kernel boot string and append this:

nvidia.NVreg_EnablePCIeGen3=1

Hopefully this will work.

6

Thanks. I created an /etc/modprobe.d/nvidia.conf file and also added the above string to grub, and now the output of lscpi for the x16 lanes is:

LnkSta:	Speed unknown, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
    LnkCap:	Port #0, Speed unknown, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <4us

Where what used to say “5GT/s” now says “unkown.” Is there another way to probe PCIe speed?

Do I need to upgrade or patch the kernel?

7

nvidia-settings shows the information you’re interested in, under GPU 0/1/2/etc.

In my case:

Bus Type: PCI Express x16 Gen2
PCIe Generation: Gen2
Maximum PCIe Link Width: x16
Maximum PCIe Link Speed: 5.0GT/s
8

nvidia-settings just complains that I’m not running the nvidia x driver (because my main display is non-nvidia).

9

Got it- installed the latest version of pciutils (not the one available through Ubuntu’s apt-get), and the speeds are reported as 8GT/s. Thanks for everyone’s help.

10

OK, here’s a different solution which should work no matter what.

Update your kernel boot string and append this:

  1. nvidia.NVreg_EnablePCIeGen3=1

Hello:
I did that for PCIe 3.0 certified Gigabyte X79-UD3. ivy bridge i7-4930, two GTX-680, GNU Linux Debian amd64 jessie, nvidia driver 319.60. I believe correctly, by:

  1. typing ‘e’ at grub prompt,
  2. adding the option to the linux line,
  3. Ctrl-x to boot

as shown by

~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.10-3-amd64 root=/dev/mapper/vg1-root ro 1. nvidia.NVreg_EnablePCIeGen3=1 quiet
~$

Nothing improved, lspci -vvvv showed both LnkSta and LnkCap at 5GT/s, as for PCIe 2.0, and established molecular dynamics simulations run at the same speed as at PCIe 2.0.

I wonder whether the suggested nvidia string is correctly spelled, or correctly inserted into the grub’s linux line (there was no break in the line, should this appear here). Please notice that the string suggested for adding to nvidia.conf is different. I tried also this nvidia.conf suggestion, same unsuccessful move.
thanks
chien

11

Sorry for resurrecting this old thread, but I had been looking for a solution myself and stumbled upon many unanswered questions. The solutions suggested here didn’t work.

On Ubuntu 16.04, I also had to run update-initramfs (or the equivalent for your distro to update the initramfs/initrd). More details here: nvidia_367 kernel module options ignored at boot, ubuntu 16.04, /etc/modprobe.d

12

just FYI, on a 2013 motherboard, above advice didn’t work until I updated the BIOS, even though the old BIOS supported PCIe3.0 (the CPU and GPU did as well). Don’t think there’s any difference between putting the command line option into grub menu or launching manually or using an argument to modprobe - all ways propagate the option to at least the option dictionary in the module(s).