Has anyone managed to get PCIE 3.0 working with the GTX 670? What bandwidth should I expect to get? I’m using a X9DR3-F which has dual Xeon E5 processors and supposedly supports PCIE 3.0 but I’m having difficulty enabling it.
Sorry to answer my own question but it works perfectly on the ASUS RoG Maximum V Gene. I get about 12GB/s in each direction for pinned memory.
On the X9DR3-F it doesn’t work at all. You have to enable Gen3 in the BIOS which causes it to not boot. Something else to watch out for with the X9DR3-F is that although from the photographs it looks like you can install 3 video cards in this motherboard in fact they have changed the layout and one of the PCI-Express slots is now obstructed by the DIMM slots.
Impressive! Do you know how close that is to the host memory bandwidth in that system?
Just to clarify, that’s using bandwidthTest so its not testing both directions simultaneously. I’m not sure if the GTX 670 has dual DMA engines or if that’s still reserved for Quadros and Teslas. I’m running dual channel PC3-12800 (1600MHz) at stock clocks and according to SiSoft Sandra my memory bandwidth is almost exactly 20GB/s.
I’m still trying to get Gen3 working on the X9DR3-F. I’ve got a new BIOS version from Supermicro which has had some effect although my secondary video card is now not showing up in Windows and the primary one isn’t running at Gen3 speeds. One thing I’m wondering though is whether nVidia are treating this as an “unvalidated” platform. All of the marketing material from both Intel and Supermicro indicate that this combination is fully PCI Express 3.0 certified but I guess it does have more in common with X79 and Sandy Bridge-E than it does with Ivy Bridge.
Update: I applied the registry hack to bypass the restriction enforced by nVidia and my primary video card is actually now (sort of) working at Gen3 speeds. Host to device speeds are very erratic (anything from 7GB/s to 12GB/s) but device to host speeds are a solid 12GB/s.
Its hard to know whether Supermicro, Intel or nVidia is to blame for this mess.
allanmac pointed out this interesting explanation and workaround, which may relate to what you’re hitting.
Yes, the force-enable-gen3.exe is functionally the same as the registry hack I mentioned. It allows me to get (sort of) GEN3 speeds but doesn’t solve my stability problems. I guess cases like mine are why nVidia are saying this is “at your own risk”. But unlike the Sandy Bridge-E on X79 there is absolutely no ambiguity in the Intel specifications about whether the Xeon E5 on C602/C606 is PCI-Express 3.0 certified or not.