Linux PCI-E 3.0 with GTX 670

I just bought a server which was supposed to be a PCI-E Gen3 GPU compute server.
It has an Intel W2600CR motherboard with dual E5-2620 and 4 x ASUS GTX 670 cards and runs under Linux.
Of course the documentation for the motherboard says that the motherboard has the PCIE3 slots. The documentation for the ASUS GeForce GTX 670 card says that it works with PCIE3. Linux (RHEL63) sees the card running at PCIE2 speed.

I see one disclaimer on the Nvidia site re: the GTX 670:

"GeForce GTX 670 supports PCI Express 3.0. The Intel X79/SNB-E PCI Express 2.0 platform is only currently supported up to 5GT/s (PCIE 2.0) bus speeds even though some motherboard manufacturers have enabled higher 8GT/s speeds."

Okay … the W2600CR is listed as a c600-a chipset, which isn’t X79/SNB-E, but I guess based on it… Maybe this is my problem…

I tried the latest Nvidia driver, and applying the “NVreg_EnablePCIeGen3=1” module option under Linux, but when I run a simple “deviceQuery” with the module option in place, the system crashes and reboots. I guess that’s what they meant when they said that this might introduce instability!

Okay – so I don’t want instability, but I want to run 4 x ASUS GTX 670 cards in a system with dual Intel E5-2620 at PCIE3 speed. Is this possible? Certainly there is some server board that has been certified by Nvidia to work in this configuration? This is actually the second time I’ve been through this – the first time I bought a Dell server and was told that everything would work fine, but the server didn’t come with the proper power leads (even though I ordered the proper GPU kit), and couldn’t handle the 4 GTX 670 cards without removing the disk controller card – great! So now I have to get this working with this system, or change out the motherboard and get it working with something else…

Is there a place where Nvidia has certification details so that I could pick a board and know that it’s going to work in this configuration?

The crash may also be related to a board issue. I am running the X79 platform with ASUS Rampage IV boards and have found the PCI-E 3.0 support to be completely stable using the latest Linux drivers with the GEN3 parameter. I run anywhere from 32 to 40 lanes at GEN3 under 24/7 load. However, I had another EVGA x79 board where one of the slots would fail to negotiate at 8 GT/s with the NVIDIA cards. I moved the processor from the EVGA board to an ASUS board and the GEN3 problems were resolved. I suspect that some of the motherboard manufacturers have not done any extensive testing to make sure that the slots, traces, BIOS, etc can actually handle the PCI-E 3.0 spec.

Regarding PCI-E 3.0 support via the processor, your particular processor along with the rest of the E5-2600 series of processors are actually listed as supporting PCI-E Revision 3.0 via Intel’s website.