Multigpu motherboard for CUDA

Accelerated Computing CUDA CUDA Programming and Performance
1

Hi,

Can anyone recommend a good motherboard/PSU for a multi-G8x solution? As it’s for CUDA, I imagine the expense of SLI compatibility is unrequired. Any ideas?

Cheers,
Paul

2

SLI compatibility isn’t required, but AFAIK it’s usually the case that high-end SLI motherboards offer 2x16 PCI-e lanes, which of course helps your throughput to and from the cards.

3

Thanks. I am indeed finding that the two often come together (SLI and multiple PCIEx16 slots).

Looking at the Overclockers Ltd. motherboards section at [url=“http://www.overclockers.co.uk/productlist.php?groupid=701&catid=5”]http://www.overclockers.co.uk/productlist....pid=701&catid=5[/url] there are two that differ in price, with the second almost 50% more expensive.

P7N Diamond
[url=“http://www.overclockers.co.uk/showproduct.php?prodid=MB-098-MS”]http://www.overclockers.co.uk/showproduct....rodid=MB-098-MS[/url]

Striker II Formula
[url=“http://www.overclockers.co.uk/showproduct.php?prodid=MB-260-AS”]http://www.overclockers.co.uk/showproduct....rodid=MB-260-AS[/url]

I don’t mind the price difference too much, but can I in general assume that a multi-gpu motherboard will run CUDA under Linux? Are there compatibilty issues? Also, can anyone recommend a 4x(PCIex16) motherboard?

Paul

4

If you are going AMD, the MSI K9A2 AMD 790FX has 4 PCI-Express 2.0 x16 slots. I haven’t tested this board, as it was extremely sold out when I built my Phenom-based workstation for CUDA, but it looked good on paper.

Intel or AMD, you’ll want a quad-core CPU if you plan to use 4 GPUs at the same time.

5

Thanks, I’m going to order that MSI board with a Phenom and 4 GTXs. Any idea what kind of PSU wattage I’ll need? If one GTX needs 5-600W, maybe 4 will need 1500W?
Cheers,
Paul

6

I’m pretty sure you won’t be able to fit 4 GTX boards in, since the last slot is too close to the edge of the motherboard. A double-wide card will hit the bottom of your case.

I think your options are 3 x 8800 GTX vs. 4 x 8800 GT. In aggregate, that’s:

  • 3 x 8800 GTX: 384 stream processors, 260 GB/sec memory bandwidth, 185Wx3 = 555W
  • 4 x 8800 GT: 448 stream processors, 230 GB/sec memory bandwidth, 105Wx4 = 420W

Part of that power is supplied via the PCI-Express slots, and part is supplied by the extra power connectors on the card. I’m worried that 4 slots drawing 75W each from the motherboard will lead to stability problems.

tachyon_john and others have definitely built 3x8800 GTX workstations before. Search for their posts for PSU advice. The 4x8800 GT option should fit within the same power envelope.

Definitely let us know how this works out, but I’d buy your parts from a place with a good return policy. :)

7

The ASUS M3A32-MVP Deluxe/WiFi is a good board which supports 4 16x PCIe. I have it with:
Phenom 9600
8600 and 2xTesla
Running Fedora 8 (Fedora 7 SDK)
850W Cooler Master PSU

no stability problems so far.

8

That is interesting. I believe that is one of the cards I tried but couldn’t get 16x performance out of the any slots but the first. Can you post the bandwith test results from the SDK. You will have to mod the test to get the other board. I finally setled on the F790X chipset that gave the same value across all the slots.

Thanks

9

I am out of town right now, but I will do that as soon as I get back home… prob friday.

10

Use this case. Then you can put 4 dual slot cards in.

http://www.lian-li.com/v2/en/product/produ…ss_index=61&g=f

But you need bend the bracket of the fourth card.

I have a machine with MSI P6N Diamond with four dual slot cards in this case.

11

Here are the bandwidth tests for the different cards… looks like it is as expected with one Tesla running at 16x and the other at 8x (for PCIe config 16x 8x 16x 8x). The disparity is not really an issue for me at this point, but I have been considering switching to the NVIDIA 780i chipset to get 3x true 16x.

Card 1 - 8600
Host to Device Bandwidth for Pageable memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 1248.9

Device to Host Bandwidth for Pageable memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 1428.0

Device to Device Bandwidth
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 19514.0

Card 2 - Tesla #1
Host to Device Bandwidth for Pinned memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 1328.8

Device to Host Bandwidth for Pinned memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 1599.3

Device to Device Bandwidth
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 63961.6

Card 3 - Tesla #2
Host to Device Bandwidth for Pinned memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 2680.4

Device to Host Bandwidth for Pinned memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 3048.2

Device to Device Bandwidth
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 64006.4