I run four GTX 295 cards on a P6T7 motherboard with Ubuntu 9.10 64-bit (not an officially approved distribution, had to install gcc 4.3 for nvcc) and I have run the same configuration with RHEL 5.3. Both cases worked fine.
Basically, the best CUDA device configuration depends on the answers to these questions:
- Is my algorithm easy to divide between multiple devices with minimal (or no) exchange of data between them? Will the performance scaling be approximately linear in the number of devices?
If not, you would be better off with the fastest single device, the GTX 285. If multiple GPU operation would be beneficial, then the GTX 295 might be worth considering (or just more GTX 285s).
- How much memory in each device do I need to hold the data I will be operating on?
Each GTX 295 half gets 896 MB, the GTX 285 comes in 1 and 2 GB models, and the Tesla is 4 GB.
- How much data do I need to send over the PCI-Express bus? How often do I need to do this?
If the answer is “a lot of data very frequently”, then the GTX 295s will not be very helpful as each CUDA device competes with the other one for the x16 link. Moreover, unless you get a very unusual motherboard, you will max out the bandwidth at two devices. Beyond that point, you will be sharing a fixed amount of PCI-Express bandwidth with more and more CUDA devices. For bus-bound kernels, a pair of GTX 285s or Teslas would be best.
- Do I have the patience to deal with the lower quality assurance of the GeForce series? (Or conversely: Do I have the budget to afford the greater quality assurance of the Tesla series?)
I’ve used about a dozen GeForce cards since CUDA was released, and had none dead-on-arrival. One card died after a year of use. I know of no other failure stats for GeForce or Tesla cards. You might be able to get Tesla failure rates if you sign an NDA, but to be honest they probably won’t be of much use to you unless you are planning a large deployment.
If you aren’t sure what the answers to these questions are (and they are sometimes hard to figure out without something to play with), then you might want to build a solid workstation with just one GTX 285. If things look good, then you can add Tesla/285/295 cards as needed, and if things don’t work, then you just have a nice workstation with a good graphics card. :)