I just bought a server which was supposed to be a PCI-E Gen3 GPU compute server.
It has an Intel W2600CR motherboard with dual E5-2620 and 4 x ASUS GTX 670 cards and runs under Linux.
Of course the documentation for the motherboard says that the motherboard has the PCIE3 slots. The documentation for the ASUS GeForce GTX 670 card says that it works with PCIE3. Linux (RHEL63) sees the card running at PCIE2 speed.
I see one disclaimer on the Nvidia site re: the GTX 670:
"GeForce GTX 670 supports PCI Express 3.0. The Intel X79/SNB-E PCI Express 2.0 platform is only currently supported up to 5GT/s (PCIE 2.0) bus speeds even though some motherboard manufacturers have enabled higher 8GT/s speeds."
Okay … the W2600CR is listed as a c600-a chipset, which isn’t X79/SNB-E, but I guess based on it… Maybe this is my problem…
I tried the latest Nvidia driver, and applying the "NVreg_EnablePCIeGen3=1" module option under Linux, but when I run a simple "deviceQuery" with the module option in place, the system crashes and reboots. I guess that’s what they meant when they said that this might introduce instability!
Okay – so I don’t want instability, but I want to run 4 x ASUS GTX 670 cards in a system with dual Intel E5-2620 at PCIE3 speed. Is this possible? Certainly there is some server board that has been certified by Nvidia to work in this configuration? This is actually the second time I’ve been through this – the first time I bought a Dell server and was told that everything would work fine, but the server didn’t come with the proper power leads (even though I ordered the proper GPU kit), and couldn’t handle the 4 GTX 670 cards without removing the disk controller card – great! So now I have to get this working with this system, or change out the motherboard and get it working with something else…
Is there a place where Nvidia has certification details so that I could pick a board and know that it’s going to work in this configuration?