Quad GTX280 Configuration

Hello All,

Sorry for the cross post. I originally put this on the wrong forum.

I would like to set up a quad GTX280 configuration similar to the what the FASTRA guys put together: FASTRA specs. Any suggestions on modifying this config would be greatly appreciated. My application is very calculation intensive with relatively few data transfers so transfer speed is a plus but not a deal breaker. My budget is $3500 and my current config is $3200 on Newegg so I think I can get this done. Please be gentle, I am not a hardware guy.

I mostly emulated the Fastra config according to their website:
AMD Phenom 9850 processor + Scythe Infinity CPU cooler
4x EVGA GTX 280
4x 2GB Corsair Twinx DDR2 PC6400 memory
MSI K9A2 Platinum motherboard
Samsung Spinpoint F1 750GB HDD
2x Silverstrone DA1000 1000W PSU
Lian-Li PC-P80 Armorsuit case ??Cooling issues??
Windows XP 64-bit

Of course, I changed their 9800’s to GTX 280’s. I went with 2 x Sliverstone 1000W PSUs instead of their ThermalTake 1500W. I am also looking to change the case to something with better airflow. Air cooling only for this setup is preferred. Regarding to mobo, I would prefer Intel but I don’t think I have any options that can handle 4 GTX280s.

Let me know what you think.

Thanks,
Mike

9800 X2s are gonna be more powerful than GTX 280s. Some time in the near future the GT200’s version of the X2 will come out and would be the foundation of the next-gen FASTRA.

Also, there’s these crazy things called PCIe port replicators. You could technically connect like a dozen cards to one of these things, if you really want to one-up those Belgians.

PS. you can use two power supplies, but one of them you’re gonna have to turn on with a paperclip.

I’d appreciate if you could show links on the PCIe port replicators
and how connect two power supplies.

i use 2 power supplies.

but mine is a little different.

i have my external PSU (that is just a normal psu that normally goes inside the computer) to power 4 sata hard drives and a normal internal sata burner…

it looks funny cause i took the normal hard drive rack (was riveted) that is usually inside the computer box and removed it and put my 4 hard drives in and it is just laying on my floor close to my computer.

the psu i use on the outside is sitting right next to it with the dvd burner on top

then i have a antec 500 watt psu that is in my computer box powering an nf4 asus a8n32-sli-deluxe with 4 sticks of 512MB ram and an amd 64 x2 4200 OC to 2.65ghz with 2 8800gts g92s and a zalman cnps 9700 fan.

never have a power problem…it all is rock solid.

i can take a screen shot if you want.

you dont have to use a paperclip (i use the paperclip also) but there is a better method where you take a molex connector with the psu that is inside your computer and connect 2 wires that goto the big 20/24pin connector on psu2 (outside the computer) the 2 wires you connect from the molex connector to the 20/24pin connecter are the PS ON or the power supply on.

i forget what pin numbers they are but this is the proper way so when you tell your computer to shutdown or restart the molex from psu 1 sends the signal to the 20/24 pin on the psu2

it is the proper way but i just use the paper clip method but my drives never shut off when i tell the computer to shutdown or restart.

:)

like i say it all works rock solid.

http://www.helpwithpcs.com/courses/power-s…psu-pinouts.gif

PS_ON has to be shorted to a GROUND for an ATX psu to start up. Plus, you may need to put a little load (0.1-0.5A) on the various voltages. (Although I think the best PSUs don’t have this requirement.) Read the manual carefully.

Re: PCIe port multipliers:

Searching for “pci express expander” gives a couple results. These are typically used for RAID cards but they can hold anything. The ones I saw have 7 single-width slots (ie, they are designed to fit into ATX chasses).

The PCIe expanders are all based around these chips, made by PLX: http://www.plxtech.com/products/expresslane/switches.asp

The problems is that a) the ready-made expanders are freakin expensive (The PRO EX is $3.1k at powermax, the Fusion RAID is $4k) b ) they don’t have that many slots, especially counting double-wide cards and c) they don’t use the top-of-the-line PLX chips.

The totally uber thing to do would be to take the PEX 8648 chip and make one yourself. It would host 10 GTX280s (and you could connect four of these monsters to a single PC). I’m salivating already. It’s actually not that hard to make the PCB with the one chip, probably easier than making the chassis. It would be crazy fun, and perhaps profitable (and the market is apparently bigger than just CUDA). Of course, this would make sense for apps where neither host-card bandwidth nor CPU are the bottlenecks. Otherwise you’d recode your app in MPI and make a real cluster.

Thanks for the information.

Thanks for all the replies. Below are some updated specs on my CUDA box. My application is calculation intensive / low data transfer so I am willing to accept some bottlenecks in my PCI-e data transfers.

I do have a couple of questions for the hardware experts:

  1. Ballpark what kind of transfer speeds could I expect with the config below?

  2. Apparently the quad GTX 280 config is challenging because of the 6 pin + 8 pin requirement versus the 2 x 6 pin requirement. Is this a PSU issue or a mobo issue? I am getting 2 PSUs both with 2x6 and 2x8 connectors. Will that fix it? If not, I will move to the GTX 260.

  3. There is a GTX 260 on Newegg with 216 cores. I thought is was 192? Can you over-core ( like over-clocking) a card? If so, I assume I should not use a modified card for CUDA.

Here are my current specs. The following config comes in at $2250 on NewEgg so I have about $1k of wiggle room for any meaningful upgrades :

4x EVGA GTX 260
Foxconn Destroyer motherboard (integrated video)
AMD Phenom 9850 processor
Xigmatek HDT-S1283 cooler
4x 2GB Corsair Twinx DDR2 PC6400 memory
Samsung Spinpoint F1 750GB HDD
2x Antec TPQ-850 850W
Antec 1200 case
Windows XP 64-bit

Let me know if anyone has further suggestions.

Thanks,
Mike

Change the CPU up to a Phenom 9950 Black edition. It’s only $10 more than the 9850. Yes, it’s only a 100Mhz boost, but that could still be useful to reduce potential CPU bottlenecks.

Heh, checking on Newegg, for some reason this week the 9950 Black is $174, and the plain 9850 is $209. I think your choice is easy!
Buy the retail box (not OEM) for the warranty, it’s only $5 difference.

TDP on 9950 is significantly higher (20W, I believe).

The 9800 GX2’s will give you more performance. By my count, 50% more (although maybe you’d get less efficiency out of 8 cards vs 4). If you wait some months (i’m guessing 3), the gt200 GX2s will come out. (maybe you can buy 4 cheap cuda cards now, write your code, and then use saved cash + extra $1k to get four of the new cards?)

Thanks for all your help. I pulled the trigger and now we’ll see if I can assemble this thing correctly.

  1. Processor - Thanks, I upgraded to the 9950, they actually have a lower power 125W version now and it was cheaper than the 9850 when bundled with Ram

  2. Ram - yeah, I probably could have gone a bit cheaper but they had a $25 rebate so it ended up begin $45 for a 2x2gb set.

  3. Cards - Ok this was a tough one. I went with the GTX 280’s. I know I may suffer compared to the 9800GX2 but I do want a 200 series and I may need a good portion of that extra RAM. The 8pin/8pin power requirement is resolved by the fact I am using two PSUs. Anyway, I am starting with one GTX280 for testing and by the time I finish, who knows, there maybe some updgrade to consider.

  4. The Foxconn is going to have some bandwidth problems. I know this but I am hoping it will be resolved by the time I move to 4 cards. If not, I will have to make some changes.

So here is where I ended up. About $3k if I bought all the cards. About $1.5k with one card. Not bad for the processing potential:
4x EVGA GTX 280
Foxconn Destroyer motherboard (integrated video)
AMD Phenom 9950 processor 125W
Xigmatek HDT-S1283 cooler
4x 2GB Corsair Twinx DDR2 PC6400 memory
Samsung Spinpoint F1 750GB HDD
2x Antec TPQ-850 850W
DVD-RW drive
Antec 1200 case
Windows XP 64-bit

Let me know if anyone has any BIOS, memory or card setting suggestions.

Cheers,
Mike

Rather than trying to hack a second external power supply, try one of the dedicated GPU power supplies that fit in a 5.25" drive bay. These provide just +12V for graphics cards, providing two 6-pin and two 8-pin PCIe power connectors. We’re starting to use these to upgrade older boxes that have an open slot but don’t have a big enough power supply or enough connectors.

Thermaltake W0157RU 450W $70
ePower EP-350CD 350W $90
FSP Group BoosterX 5 450W $90

Great idea! I did not know these existed. Much cleaner solution. The nvidia site says the GTX 280 takes 236watts per card - I will put 2 cards on the supply. I don’t know anything about power but it seems I should go with the Thermaltake 650W instead of the 450W. It takes two drive bays but I have the room. So that’s two GTX 280’s on the Thermaltake VGA PSU and everything else on the Antec 850W PSU. Seems like enough power to go around all in one case. I will need to check the cooling but it this could work. THANKS!