I’ve had my eye on this idea too (PhD student in computation biology). There are a few issues, most of which have been covered here with the same theoretical solutions as I found.
One extra problem though, on most motherboards the bottom two PCI-E slots are too close to plug directly into (since they are designed for 7-slot cases). One possible way around that would be to use a PCI-E riser ribbon card like this one: [url=“Ably-Tech Corp. == View Products”]Ably-Tech Corp. == View Products possibly with a straight riser on the card above too.
From my point-of-view, this certainly seems like a very feasible build, and as soon as I can persuade someone to give me enough money for it, I’d build one :)
If I can add my two cents worth. Apart from the upcoming multi core CPU’s, I was just reading a review of a product in a gaming magazine. There are motherboards now being released that support two physical CPU chips, i.e. 2x 4 cores would give your 8. I belive the board is based on Xenon technology.
There is another issue as well - heat. I’m using two 9800GX2 in a system, and despite of heavily modifications of the chassis (read: lots of big fans), it’s still a problem. With four, I guess you have use a water cooling system.
andrewgerm there have been motherboards that support two cpu’s out for a while, now intel has released skulltrail which uses xeon as you say, but allows for SLi or crossfire.
the main issue you guys will face is the fact that on all motherboards the bottom 2 pci-e slots are not spaced to allow a double slot card to fit on both, you will need a riser adapter and some custom stuf to fit it all on the motherboard let alone in a case that isnt just a cardboard box :P
Skulltrail! Thank you. Thought it had something to do with castles and knights.
The one issue would be that the multi GPUs and CPUs scale nicely. I’d guess you’d eventually reach a point of negligable returns, when the overhead of controlling all the processors becomes too much of an overhead.
Looking forward to hearing about the final results.
I do a lot of video editing, so am also looking into water cooling. A bit expensive, but well worth, not just for the noise (which causes problems when doing sound work) but when running renders, my system flatlines at almost 100% CPU usage, and that for several hours has a marked effect on temperature, not to mention when the GPU kicks into high gear, power consumption.
A single 9800 GX2 appears as two different CUDA devices, each with 128 stream processors. It really is two cards glued together with a PCI-Express bridge in between.
@jarjar
You will need a power supply that has an 8pin PCI-E power connector. Unless you purchased your 550W power supply in the past 6-9 months, its very unlikely that it has an 8pin PCI-E power connector.
Congrats! That is quite a monster, and hopefully you’ll be able to get some additional forced air cooling on that thing before you burn up your cards at 100C. :) (BTW, you might want to write some automated testing kernels to make sure that you aren’t getting silent GPU memory corruption at that temperature. It’s been observed at temperatures as low as 90C before.)
I’m curious how well you keep the 8 GPUs busy with your application. Does only having 4 CPU cores for 8 GPUs limit your efficiency? Can you run benchmarks where you enable 1, 2, 3,…,8 GPUs at a time and see how linear the performance increase is?
So far, we have not noticed any memory curruption. For normal use, we don’t overclock the GPUs and they are running stable around 92C (which is still hot, I agree:) But we will certainly keep monitoring correctness of the computations.
Regarding the benchmarks: although we haven’t made a nice graph yet, I can report that for our application the performance scales about linearly with the number of GPUs. There appears to be no performance hit from using two threads per CPU core. Our program is subdivided into rather large kernels, so that the time overhead of kernel launches is small and there is no need for very intensive polling to see if one of the GPUs is finished.