Any future problems running GPUs for 12+ hours at a time? running cards for long periods of time whi

[SPWorley]

SPWorley- Can you refresh our memory on what mobo you are using in this setup? Thanks, V.

I use a variety. But I’d most recommend the very cheap MSI G65, which gives you 3 double-wide x16 PCIe slots and also includes embedded NV display GPU, so you don’t have to waste a full GPU for display. Pop in a cheap AMD hexacore, and you get a really efficient system.

I also have a couple systems based on the ASUS i7 MBs, but I now like the easier 3-GPU G65 motherboard.

[seibert]

I did talk to a system builder who was incredulous that you’d ever run consumer cards 24/7. His point was that failures were rare, but clearly way more common if stressed with 24/7 use. The professional cards (Tesla) run cooler (they have lower clocks), and have been pre-tested more thoroughly, so you’ll have many many fewer failures than consumer cards when stressed like that.

The interesting point was the economics of board failure. One dead board will crash a production machine, perhaps it’s on one machine in a big server farm. But now you just interfered with whatever compute you’re doing and much throughput computes aren’t robust to failure, and often need manual intervention, and/or can cause the whole server farm to wait, so it becomes a big reliability issue since basically the one board failure can cause big issues with your entire farm. And that’s with a catastrophic failure… a transient failure is even worse.

And finally, if you do have a failure, you now need the skill and experience to find and replace the bad board (which seems trivial, but it’s a problem as machine counts grow!) and that replacement costs money and time.

So his final point was that it was well worth paying the significant extra cost for a Tesla for the simple reason that the higher MTBF reduced these stresses and costs.

This isn’t exactly what the original poster was asking but I mention it here because it’s interesting. Frankly I haven’t had any problem with using consumer cards 24/7. My only hardware failures were with a bad UPS and with a bad PSU.

This certainly makes sense with a large deployment and if the cost of downtime is high. For smaller scale academic applications the reliability features of Tesla are seldom worth the cost by themselves. I could burn out a GTX 470 every 6 months and still be financially ahead of a Tesla C2050 with a deployment lifetime of 3 years.

Incidentally, I totally believe the statement about one bad device taking out a large cluster. That would explain the terrible uptimes I’ve seen in most > 500 node systems I’ve worked with. Really, it demonstrates that we need come up with ways build more fault-tolerance into our distributed compute jobs. Having to assume flawless performance of thousands of devices is driving up the cost of HPC. :)

[seibert]

I did talk to a system builder who was incredulous that you’d ever run consumer cards 24/7. His point was that failures were rare, but clearly way more common if stressed with 24/7 use. The professional cards (Tesla) run cooler (they have lower clocks), and have been pre-tested more thoroughly, so you’ll have many many fewer failures than consumer cards when stressed like that.

The interesting point was the economics of board failure. One dead board will crash a production machine, perhaps it’s on one machine in a big server farm. But now you just interfered with whatever compute you’re doing and much throughput computes aren’t robust to failure, and often need manual intervention, and/or can cause the whole server farm to wait, so it becomes a big reliability issue since basically the one board failure can cause big issues with your entire farm. And that’s with a catastrophic failure… a transient failure is even worse.

And finally, if you do have a failure, you now need the skill and experience to find and replace the bad board (which seems trivial, but it’s a problem as machine counts grow!) and that replacement costs money and time.

So his final point was that it was well worth paying the significant extra cost for a Tesla for the simple reason that the higher MTBF reduced these stresses and costs.

This isn’t exactly what the original poster was asking but I mention it here because it’s interesting. Frankly I haven’t had any problem with using consumer cards 24/7. My only hardware failures were with a bad UPS and with a bad PSU.

This certainly makes sense with a large deployment and if the cost of downtime is high. For smaller scale academic applications the reliability features of Tesla are seldom worth the cost by themselves. I could burn out a GTX 470 every 6 months and still be financially ahead of a Tesla C2050 with a deployment lifetime of 3 years.

Incidentally, I totally believe the statement about one bad device taking out a large cluster. That would explain the terrible uptimes I’ve seen in most > 500 node systems I’ve worked with. Really, it demonstrates that we need come up with ways build more fault-tolerance into our distributed compute jobs. Having to assume flawless performance of thousands of devices is driving up the cost of HPC. :)

[Sarnath]

Hey, Just for knowledge sake – how do you control the speed of GPU fans and temperature? I could not find any such options in nvidia-smi tool. I am not graphics savvy… so, pls bear with me,

Thanks for any info,

[Sarnath]

Hey, Just for knowledge sake – how do you control the speed of GPU fans and temperature? I could not find any such options in nvidia-smi tool. I am not graphics savvy… so, pls bear with me,

Thanks for any info,

[carl_lazlo]

Hey, Just for knowledge sake – how do you control the speed of GPU fans and temperature? I could not find any such options in nvidia-smi tool. I am not graphics savvy… so, pls bear with me,

Thanks for any info,

I first installed the EVGA Precision software. After starting the software I double click the display in the middle to break it out into the larger floating display. If you then right click on the floating display there is a tab in there for fan profile. It’s actually pretty cool, you can click and add points and drag the line to make your own cooling profile. I set mine up more logarithmic so once the temps get about 65c or so the fans ramp up in speed quickly. So far the cards never get above 72c and run about 90% of the time at 65c under 99% load.

Another thing I noticed is with the EE form factor EVGA cards the case temp does not affect the card temp as dramatically since the cards are completely enclosed and pass the heat out of the back. Obviously if the internal air temp the cards are drawing in is cooler the cards will be slightly cooler but I have not noticed as large of a difference compared to other cards I have had in the past where all of the airflow stays in the case.

[carl_lazlo]

Hey, Just for knowledge sake – how do you control the speed of GPU fans and temperature? I could not find any such options in nvidia-smi tool. I am not graphics savvy… so, pls bear with me,

Thanks for any info,

I first installed the EVGA Precision software. After starting the software I double click the display in the middle to break it out into the larger floating display. If you then right click on the floating display there is a tab in there for fan profile. It’s actually pretty cool, you can click and add points and drag the line to make your own cooling profile. I set mine up more logarithmic so once the temps get about 65c or so the fans ramp up in speed quickly. So far the cards never get above 72c and run about 90% of the time at 65c under 99% load.

Another thing I noticed is with the EE form factor EVGA cards the case temp does not affect the card temp as dramatically since the cards are completely enclosed and pass the heat out of the back. Obviously if the internal air temp the cards are drawing in is cooler the cards will be slightly cooler but I have not noticed as large of a difference compared to other cards I have had in the past where all of the airflow stays in the case.

[Nighthawk13]

I am sceptical about higher fan speed because of dust: More air throughput => More dust transported => cooling fins clogged by dust sooner.
If the fan runs slower, the “dust collection rate” is also lower(i assume).
Once the cooler is completely blocked, not even 100% fan speed can cool the card under full load.

(If you can operate your cards in a dust-filtered environment or can clean them occasionally that is not a problem, of course)
But for an unserviced box “out in the wild”, i am not sure if an increased fan speed will increase MTBF. (What fails earlier on average, electronics because of temperature or cooler because of dust?)

[Nighthawk13]

I am sceptical about higher fan speed because of dust: More air throughput => More dust transported => cooling fins clogged by dust sooner.
If the fan runs slower, the “dust collection rate” is also lower(i assume).
Once the cooler is completely blocked, not even 100% fan speed can cool the card under full load.

(If you can operate your cards in a dust-filtered environment or can clean them occasionally that is not a problem, of course)
But for an unserviced box “out in the wild”, i am not sure if an increased fan speed will increase MTBF. (What fails earlier on average, electronics because of temperature or cooler because of dust?)

[seibert]

(If you can operate your cards in a dust-filtered environment or can clean them occasionally that is not a problem, of course)

But for an unserviced box “out in the wild”, i am not sure if an increased fan speed will increase MTBF. (What fails earlier on average, electronics because of temperature or cooler because of dust?)

This is a good point. Google’s published a few papers on hard drive failure rates as a function of temperature, and they’ve found in their data centers that hard drives can tolerate higher temperatures than “conventional IT wisdom” would recommend. I’d be curious about other components as well.

[seibert]

(If you can operate your cards in a dust-filtered environment or can clean them occasionally that is not a problem, of course)

But for an unserviced box “out in the wild”, i am not sure if an increased fan speed will increase MTBF. (What fails earlier on average, electronics because of temperature or cooler because of dust?)

This is a good point. Google’s published a few papers on hard drive failure rates as a function of temperature, and they’ve found in their data centers that hard drives can tolerate higher temperatures than “conventional IT wisdom” would recommend. I’d be curious about other components as well.

[Sarnath]

I first installed the EVGA Precision software. After starting the software I double click the display in the middle to break it out into the larger floating display. If you then right click on the floating display there is a tab in there for fan profile. It’s actually pretty cool, you can click and add points and drag the line to make your own cooling profile. I set mine up more logarithmic so once the temps get about 65c or so the fans ramp up in speed quickly. So far the cards never get above 72c and run about 90% of the time at 65c under 99% load.

Thank you Carl! I was more looking at an NVIDIA Tool that can collect these statistics… An utility for controlling and monitoring the card!

Anyway, Thanks a lot!

[Sarnath]

I first installed the EVGA Precision software. After starting the software I double click the display in the middle to break it out into the larger floating display. If you then right click on the floating display there is a tab in there for fan profile. It’s actually pretty cool, you can click and add points and drag the line to make your own cooling profile. I set mine up more logarithmic so once the temps get about 65c or so the fans ramp up in speed quickly. So far the cards never get above 72c and run about 90% of the time at 65c under 99% load.

Thank you Carl! I was more looking at an NVIDIA Tool that can collect these statistics… An utility for controlling and monitoring the card!

Anyway, Thanks a lot!