Enabling PCIe 3.0 with NVreg_EnablePCIeGen3=1 ?

Graphics / Linux Linux
1

Hello,

I’m in the process of building a machine to do some parallel computing using the cuda toolkit. The specification of the machine is:

Motherboard: MSI-X79G45(8D) | Processor: Intel i7 3820 | 16Gb DDR3 Kingston Genesis 2400 | GPU1: Geforce GTX 660 (For compute only - not connected to a monitor) | GPU0: Geforce 210 (For Monitor) | OS: Scientific Linux 6.3 (64bit) - 2.6.32-279.22.1.el6.x86_64

The GTX 660 is installed in the second PCIe 3.0 slot on the motherboard which runs at x16 and the MSI BIOS has PCIe 3.0 enabled.

I have succesfully installed the latest GPU drivers: 313.18 (64bit) and have attempted to set the paramater [NVreg_EnablePCIeGen3=1] to enable PCIe 3.0 through the following process:

Set /etc/inittab to [id:3:initdefault] to disable X and restart the machine at the command prompt

Log in as [SU]

Unload the Kernel Module with [modprobe -r -v nvidia] (Linux confirms module)

Set the Kernel Parameter with [modprobe nvidia NVreg_EnablePCIeGen3=1] (No confirmation or error)

Load the Kernel Modlue with [modprobe -v nvidia] (No confirmation or error)

Reset /etc/inittab to [id:5:initdefault] and retsart the machine into X

When I open NVIDIA X Server Settings and examine the settings for GPU1 (GTX 660) I get the following:

Bus Type: PCI Express x16 Gen2
Bus ID: PCI:3:0:0
PCI Device ID: 0x11c0
PCI Vendor ID: 0x10de
IRQ: 40

PCIe Generation: Gen2
Maximum PCIe Link Width: x16
Maximum PCIe Link Speed: 5GT/s

I was expecting to see Maximum PCIe Link Speed as 8GT/s not 5GT/s?

Does anyone know where I am going wrong? I’m new to Linux so any advice would be a real help.

Thanks in advance.

2

I compiled the 313.18 drivers this evening with kernel 3.7.9. My cards are reporting as running at 8 GT/s via lspci and the NVIDIA bandwidthTest application confirms that bandwidth is nearly double that of PCI-E 2.0:

GPU 0: LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
GPU 1: LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
GPU 2: LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

With the GPU under load, can you try running the below command to check the link speed:

lspci -vv -s 03:00.0 |grep LnkSta

3

I did some more research into this. The 8.0 GT/s PCI-E reporting I believe was added to a later kernel version than the one you are running. I checked the 2.6.32 kernel source and was not able to find a reference to the updates that add the 8.0 GT/s reporting. You may need to upgrade to a newer kernel. Here is a link to the patch:

[url]An Interview With Linus Torvalds: Linux and Git - Part 1 | Tag1 Consulting

4

Thanks for your advice. I’ll attempt to patch the kernel as you suggest (another first for me!) and then check the bandwidth. I’ll post an update on how it goes.