Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Categories

In this Discussion

Tags in this Discussion

Max number of GPU supported per host?
  • Vote Up0Vote Down hammer256
    Posts: 4
    I've recently converted my simulation to support multiple CUDA GPUs, and I'm wondering, what is the max number of GPUs supported by the driver for the host computer? My algorithm should be able to handle up to 32 GPUs.
    I'm running Linux 64 bit (Gentoo) with CUDA 4.1, currently with a GTX 470 and a GTX 580 (hardware we had lying around in the lab) on a core i7-2600k host. I only transfer KBs of data over pci-e per time step in the simulation, so pci-e bandwidth is not a concern. I'm thinking about building a 8 (or 16, using expanders) GPU computer, so I want to know what the driver's limitations are for number of GPUs.

    Thanks,
    Wen
  • 5 Comments sorted by
  • Vote Up0Vote Down mcleary
    Posts: 49
    I think that the main limitation could be the slots that your motherboard has. The max number of GPUs per host I know for mainstream computers is four.
  • Vote Up0Vote Down fwende
    Posts: 19
    since you can buy gpu cluster nodes with up to 8 tesla gpus installed, the driver shouldn't be the problem.

    by the way: don't use consumer components for such a system, as on both layers (host and gpu) you won't have support for ecc. in the past i made bad experiences with consumer gpus which were going to produce faulty output when board-/gpu-temperatures were too high (i guess signal inconsistencies [encouraged by high temperatures] which caused effects comparable to soft-errors; gamers refer to this as 'pixel-faults'). in a system with more than 8 gpus installed you need a smart and efficient cooling solution. keep that in mind when starting your project:-)
  • Vote Up0Vote Down hammer256
    Posts: 4
    Yeah good point. I've not run into memory issues yet, but I can imagine as I use more GPUs and increase thermal load things are going to have a much higher probability of failure... Those crazy water cooling blocks look mighty tempting now. Or I should just bite the bullet and actually get a proper Tesla node. Either way I really want to wait for Kepler to come out first.

    But do you think the driver can support more than 8 GPUs?
    Thanks a lot :)
  • Vote Up0Vote Down fwende
    Posts: 19 Accepted Answer
    hi again,

    yes, the driver can. for instance, see the fastra2 project from university antwerpen. they came up with a system using 7 gtx295 gpus installed on an asus motherboard. amongst some trouble with temperatures, they ran into problems with the 32-bit bios of the motherboard which did not allow for all these gpus run simultaneously. asus then provided a modified bios. this was 2009. maybe today's motherboards use 64-bit bios(es). as you see, it could become a challenge building up a gpu workstation/cluster(-node) using consumer components if it should contain more gpus than actually possible.

    maybe you should wait for kepler, and then buy 8 tesla gpus.

    question: what is your cpu doing when the gpus are working?
  • Vote Up0Vote Down hammer256
    Posts: 4
    That's great. Exactly the information I need. Thanks a lot.

    I try to schedule the work so the CPU is also crunching numbers while the GPU's busy. As is I get almost a 2x (something like 1.97x) speedup when I put in 2 GPUs, So at least for now the CPU is not really the bottleneck. I'll need to see what happens when I get more than 4 GPUs.

    But yeah, I'm excited about Kepler, should be pretty nice. My current neural network simulation actually doesn't require more than 2 GTX 580's to run in real time, but I'm thinking about a 10x expansion in cell numbers, and that will really benefit from 8 or more GPUs. The ultimate goal is actually 100x the number of cells I have right now, but it's not clear if my current algorithm will scale that well to those numbers.
Add a Comment