Enabling PCIe 3.0 with NVreg_EnablePCIeGen3=1 ?

Hello,

I’m in the process of building a machine to do some parallel computing using the cuda toolkit. The specification of the machine is:

Motherboard: MSI-X79G45(8D) | Processor: Intel i7 3820 | 16Gb DDR3 Kingston Genesis 2400 | GPU1: Geforce GTX 660 (For compute only - not connected to a monitor) | GPU0: Geforce 210 (For Monitor) | OS: Scientific Linux 6.3 (64bit) - 2.6.32-279.22.1.el6.x86_64

The GTX 660 is installed in the second PCIe 3.0 slot on the motherboard which runs at x16 and the MSI BIOS has PCIe 3.0 enabled.

I have succesfully installed the latest GPU drivers: 313.18 (64bit) and have attempted to set the paramater [NVreg_EnablePCIeGen3=1] to enable PCIe 3.0 through the following process:

Set /etc/inittab to [id:3:initdefault] to disable X and restart the machine at the command prompt

Log in as [SU]

Unload the Kernel Module with [modprobe -r -v nvidia] (Linux confirms module)

Set the Kernel Parameter with [modprobe nvidia NVreg_EnablePCIeGen3=1] (No confirmation or error)

Load the Kernel Modlue with [modprobe -v nvidia] (No confirmation or error)

Reset /etc/inittab to [id:5:initdefault] and retsart the machine into X

When I open NVIDIA X Server Settings and examine the settings for GPU1 (GTX 660) I get the following:

Bus Type: PCI Express x16 Gen2
Bus ID: PCI:3:0:0
PCI Device ID: 0x11c0
PCI Vendor ID: 0x10de
IRQ: 40

PCIe Generation: Gen2
Maximum PCIe Link Width: x16
Maximum PCIe Link Speed: 5GT/s

I was expecting to see Maximum PCIe Link Speed as 8GT/s not 5GT/s?

Does anyone know where I am going wrong? I’m new to Linux so any advice would be a real help.

Thanks in advance.

I compiled the 313.18 drivers this evening with kernel 3.7.9. My cards are reporting as running at 8 GT/s via lspci and the NVIDIA bandwidthTest application confirms that bandwidth is nearly double that of PCI-E 2.0:

GPU 0: LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
GPU 1: LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
GPU 2: LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

With the GPU under load, can you try running the below command to check the link speed:

lspci -vv -s 03:00.0 |grep LnkSta

I did some more research into this. The 8.0 GT/s PCI-E reporting I believe was added to a later kernel version than the one you are running. I checked the 2.6.32 kernel source and was not able to find a reference to the updates that add the 8.0 GT/s reporting. You may need to upgrade to a newer kernel. Here is a link to the patch:

http://kerneltrap.org/mailarchive/git-commits-head/2010/2/26/23477/thread

Thanks for your advice. I’ll attempt to patch the kernel as you suggest (another first for me!) and then check the bandwidth. I’ll post an update on how it goes.