We are in the process of writing a specification for a CUDA development cluster (computer aided engineering applications and multiobjective optimisation). Our budget is approximately £2500 ($4,000) which we’re expecting to cover motherboards, GPUs, CPUs and hard drives and all of the RAM if possible.
We have an old four drawer filing cabinet we are planning on converting to house the hardware. Realistically we’d like to populate two drawers with at least 2 nodes in each so 4 motherboards + hard drives + cpus in total with 4 GPUs.
Our requirements at the moment are:
Precision - the system needs good double precision performance due to the type of work.
Flexibility - we’d like to be able to investigate the performance of multi GPU configurations.
Upgrade - we’d like to have the option of installing more RAM, better GPUs and better CPUs at a later date.
Networking - would probably have to start with gigabit ethernet but ultimately like to move to infiniband.
We have compiled the following specification as a start. Any comments, advice or feedback would be much appreciated!:
Motherboard: ASUS P7P55 LX
Product code: ASUOA06PVW
Unit price: £84.59
Price for 4: £338.36
CPU: Intel i5-650
Product code: ITOA06T94
Unit price: £148.04
Price for 4: £592.16
GPU: ASUS GF GTX 460 1GB
Product code: ASUOA077CE
Unit price: £186.81
Price for 4: £747.24
Hard Drive: WD Caviar Blue 250GB
Product code: WDGA054Q5
Unit price: £31.71
Price for 4: £126.84
RAM: DDR3 SDRAM 1333MHz 2GB
Product code: WDGA054Q5
Unit price: £36.41
Price for 4: £145.64
Power supplies, switches and network cards are outside the scope of this budget so no need to include them for now. I believe this system would allow us to look at 4 CPUs/motherboards with a single GPU each or 1 CPU using 4 GPUs on the same motherboard (and any configuration in between). Am I correct?
Total so far comes to £1950.24 so some scope in the budget for upgrading GPUs/CPUs/RAM… Better to upgrade to 4 GTX 470s? i7 cpus?
The GTX470 is probably the best “bang-for-buck” consumer card at the moment. What you host the cards in, and how many hosts you need depends a lot on your envisaged usage.
We have a “whitebox” cluster with 16 nodes - each has a multicore CPU and a single GPU connected by gigabit ethernet. Typically we have 3-4 simultaneous users and most of our usage either falls into many small, single GPU jobs, or large, many node MPI based domain decomposition type simulations, and having more nodes is good for both scenarios. If you are mostly planning to have a single user, or your code is very GPU heavy and multi-GPU friendly, and host CPU light, then fewer nodes and more GPUs per host is probably preferable.
The GTX470 is probably the best “bang-for-buck” consumer card at the moment. What you host the cards in, and how many hosts you need depends a lot on your envisaged usage.
We have a “whitebox” cluster with 16 nodes - each has a multicore CPU and a single GPU connected by gigabit ethernet. Typically we have 3-4 simultaneous users and most of our usage either falls into many small, single GPU jobs, or large, many node MPI based domain decomposition type simulations, and having more nodes is good for both scenarios. If you are mostly planning to have a single user, or your code is very GPU heavy and multi-GPU friendly, and host CPU light, then fewer nodes and more GPUs per host is probably preferable.
Maybe i5-760(quad core) instead of i5-650(dual core): If you plan to upgrade to 2 GPUs later, 2 cores will be pretty busy feeding the GPUs with little reserves for calculations in CPU threads.
I second the “GTX470 instead of GTX460” for computational purposes.
If on a budget, rather use a slower dualcore-CPU and a 470.
Maybe i5-760(quad core) instead of i5-650(dual core): If you plan to upgrade to 2 GPUs later, 2 cores will be pretty busy feeding the GPUs with little reserves for calculations in CPU threads.
I second the “GTX470 instead of GTX460” for computational purposes.
If on a budget, rather use a slower dualcore-CPU and a 470.
If you are interested in future upgradeability by adding GPUs to existing nodes, I would stay away from the LGA-1156 socket processors, like the Core i5 you mention. Processors using that socket have a PCI-Express 2.0 x16 bus integrated right into the CPU and use a much slower link to the chipset, giving you the possibility of full bandwidth to only one GPU. Other sockets, like AM3 (from AMD) and LGA-1366 use a higher bandwidth link between the CPU and the chipset hosting the PCI-E bus, giving you the possibility of full bandwidth to two GPUs.
If you are interested in future upgradeability by adding GPUs to existing nodes, I would stay away from the LGA-1156 socket processors, like the Core i5 you mention. Processors using that socket have a PCI-Express 2.0 x16 bus integrated right into the CPU and use a much slower link to the chipset, giving you the possibility of full bandwidth to only one GPU. Other sockets, like AM3 (from AMD) and LGA-1366 use a higher bandwidth link between the CPU and the chipset hosting the PCI-E bus, giving you the possibility of full bandwidth to two GPUs.
Oh, and you should not install 4 GPUs on the motherboard you list above. There is neither bandwidth nor physical slots for the cards. Four GPUs is actually quite tricky, so you should not design for that unless you really need to. Two GPUs is pretty easy, and three is not too hard with some care in parts selection.
For cheaper double GPU systems, I would personally go AMD, and for a three GPU system, Core i7 socket 1366.
Oh, and you should not install 4 GPUs on the motherboard you list above. There is neither bandwidth nor physical slots for the cards. Four GPUs is actually quite tricky, so you should not design for that unless you really need to. Two GPUs is pretty easy, and three is not too hard with some care in parts selection.
For cheaper double GPU systems, I would personally go AMD, and for a three GPU system, Core i7 socket 1366.
Thanks for your comments guys! I’ve been having a rethink following your comments yesterday and modified our specification.
Not sure what exactly the expected usage will be at the moment so we’re aiming for a small scale system with as much flexibility as possible to reconfigure i.e. running with no GPUs/1 GPU per board/2 GPUs per board (following seibert’s comment). Having looked at our budget and hardware costs I can’t afford to have a full Intel setup so I have gone for a hybrid system comprising 4 GTX 470s, two Intel i7-950 nodes and two AMD Athlon nodes each node having 4GB RAM (for the moment). This also means we can assess the performance between AMD and Intel. So this system should now give us quite a bit of flexibility I believe, and some scope for upgradeability in the future.
Following the comments regarding socket performance and the motherboard capabilities I have also re spec’d the motherboards (see links to motherboard specifications below).
The questions I now have are:
Are my motherboards suitable? The Intel board supports socket 1366 and sports 3 PCIe slots (2 at x16). The AMD board supports AM3 and also has 3 PCIe slots (2 at x16).
How much power can I expect each node to draw? I’m planning on purchasing 300W PSUs for each node supporting at most, now 2 cards. Will this be enough?
Intel Specification (cost for 2 nodes = £1565/$2488)
2 x Intel i7-950, 2 x ASUS P6X58D-E, 2 x 250GB HD, 2 x 2GB 1333MHz DDR 3 SDRAM, 2 x GTX 470
AMD Specification (cost for 2 nodes = £1076/$1710)
2 x AMD Athlon 635, 2 x ASUS M4A88TD-V EVO ATX, 2 x 250GB, 2 x 2GB 1333MHz DDR 3 SDRAM, 2 x GTX 470
Thanks for your comments guys! I’ve been having a rethink following your comments yesterday and modified our specification.
Not sure what exactly the expected usage will be at the moment so we’re aiming for a small scale system with as much flexibility as possible to reconfigure i.e. running with no GPUs/1 GPU per board/2 GPUs per board (following seibert’s comment). Having looked at our budget and hardware costs I can’t afford to have a full Intel setup so I have gone for a hybrid system comprising 4 GTX 470s, two Intel i7-950 nodes and two AMD Athlon nodes each node having 4GB RAM (for the moment). This also means we can assess the performance between AMD and Intel. So this system should now give us quite a bit of flexibility I believe, and some scope for upgradeability in the future.
Following the comments regarding socket performance and the motherboard capabilities I have also re spec’d the motherboards (see links to motherboard specifications below).
The questions I now have are:
Are my motherboards suitable? The Intel board supports socket 1366 and sports 3 PCIe slots (2 at x16). The AMD board supports AM3 and also has 3 PCIe slots (2 at x16).
How much power can I expect each node to draw? I’m planning on purchasing 300W PSUs for each node supporting at most, now 2 cards. Will this be enough?
Intel Specification (cost for 2 nodes = £1565/$2488)
2 x Intel i7-950, 2 x ASUS P6X58D-E, 2 x 250GB HD, 2 x 2GB 1333MHz DDR 3 SDRAM, 2 x GTX 470
AMD Specification (cost for 2 nodes = £1076/$1710)
2 x AMD Athlon 635, 2 x ASUS M4A88TD-V EVO ATX, 2 x 250GB, 2 x 2GB 1333MHz DDR 3 SDRAM, 2 x GTX 470
You need a PSU which can supply at least 18.75A@12v per GTX470 class GPU. For a single GPU node that probaby means a 500W PSU, and for a dual GPU node, a 750W PSU.
You need a PSU which can supply at least 18.75A@12v per GTX470 class GPU. For a single GPU node that probaby means a 500W PSU, and for a dual GPU node, a 750W PSU.
Should have thought about that question before I asked it really! OK, I’ll look around for some decent 750W PSUs.
Found these on ebay so looks like we should be able to get 4 for under £100/$167…
I’m presuming that you think the motherboards look ok? Will go ahead and get a purchase order raised in the next few days to get the bulk of the hardware ordered…
Now just to fit it all into my cabinet!
Very excited about getting cracking with CUDA now and seeing the difference between GPU and CPU performance for myself! :)
Should have thought about that question before I asked it really! OK, I’ll look around for some decent 750W PSUs.
Found these on ebay so looks like we should be able to get 4 for under £100/$167…
I’m presuming that you think the motherboards look ok? Will go ahead and get a purchase order raised in the next few days to get the bulk of the hardware ordered…
Now just to fit it all into my cabinet!
Very excited about getting cracking with CUDA now and seeing the difference between GPU and CPU performance for myself! :)
Your most crucial system component for driving CUDA GPUs is the PSU. Junk PSUs will kill you. They can even physically kill your cards.
Even if they don’t fail, they can do worse and just underperform. This is most evil because it may corrupt your computations and you’d never know, leaving you frustrated and chasing phantom bugs. (I’ve posted my own anecdotes about a bad UPS causing GPU errors!)
I’d much rather have a stupid slow CPU and no-name motherboard than a cheap PSU.
Since you’re trying to get the most bang for your buck, you won’t like my suggestion, but I’d recommend the Thermaltake Toughpower series. And they’re about £100 each . But solid power is the key of a stable GPU system. (And it’s so much more true when you have multi-GPU and/or hot conditions and/or 24/7 operation.)
Your most crucial system component for driving CUDA GPUs is the PSU. Junk PSUs will kill you. They can even physically kill your cards.
Even if they don’t fail, they can do worse and just underperform. This is most evil because it may corrupt your computations and you’d never know, leaving you frustrated and chasing phantom bugs. (I’ve posted my own anecdotes about a bad UPS causing GPU errors!)
I’d much rather have a stupid slow CPU and no-name motherboard than a cheap PSU.
Since you’re trying to get the most bang for your buck, you won’t like my suggestion, but I’d recommend the Thermaltake Toughpower series. And they’re about £100 each . But solid power is the key of a stable GPU system. (And it’s so much more true when you have multi-GPU and/or hot conditions and/or 24/7 operation.)
I second that. PSUs are the one area you can’t cut corners with. I have had favorable experiences with both Cheiftec and Corsair PSUs driver single and dual GT200/Fermi setups under 24/7 conditions.