I’m currently speccing out a quad GPU box that I’m putting together for a student to work on over the summer. Ideally this machine will be a quad GTX 480 (with one hell of a beefy power supply). The question is, what motherboard should I get? A popular choice in HPC, seems to the SuperMicro X8DTG-QF dual socket motherboard, though from my own experience, and many other people on the forums:
this motherboard, and others like it with dual IOH chips seem to have very poor device to host bandwidth, typically getting about 3 GB/s where as nearly 6 GB/s is obtained for host to device.
Are there any motherboards that can sustain the full bandwidth to 4 x16 slots?
Full bandwidth to 4 x16 slots simultaneously? No. The QPI and/or HT links on the CPU can’t handle that bandwidth. In fact that CPU bandwidth can’t even keep 2 x16 slots at full bandwidth. Even an ideal dual-CPU machine with PCIe hanging off each CPU could therefore not feed 4 x16 slots. And as you linked it turns out that the dual CPU boards have PCI issues.
Now the more important question is how many motherboards give you 4 x16 links which can at least individually hit full bandwidth? There aren’t too many… no chipset gives 64 lanes so the MB needs two N200 switches on it.
Most common is the one I use, the Asus P6T supercomputer.
I also use the other Asus motherboards including the P7T WS revolution (3 x16 slots) and even the cheapo consumer P6T (2 x16, 1 x4) slots.
The other very affordable and attractive route is to get the MSI AM3 board with 3 x16 slots. This is actually particularly useful because it has onboard video, so you don’t need to use one of your x16 slots for a display. I haven’t used this board yet but I may switch to it for any more crunching boxes.
I don’t know whether such a thing exists; I believe that the number of motherboards that can actually use 4 x16 cards in x16 mode are few and have heard that most only have 2 PCI controllers.
If you are looking for more motherboards that not only have 4 PCI-E x16 but can service them all at x16, there is also the Tyan S7025: 4 fully usable PCI-E 2.0 x16 slots and 2 LGA1366 slots.
Now I dont know how running protien folding simulations would compare to what you are doing on a usage level. Part of my farm is running the MSI K9A2 Platnium mobos. But those wouldnt be good for what you are trying due to PCIE speed limitations when all 4 slots are in use.
By my research the ASUS Super Computer and the EVGA 4 way sli mobo and the EVGA SR2 dual CPU 4 way sli mobo would be your best bets.