Error Weird GTX580 Boards Crash Cuda3.2!

We have have a server with 4 GTX580.
When running a benchmark with simple cuda matrix-multiplication on all 4 boards, it crashes after 1/2 hour.

-cudaSetDevice() fails,
-calls to nvidia-smi hangs

less /var/log/messages | grep ‘NVRM’

Jul 5 22:16:56 guppy4 kernel: [2259437.389739] NVRM: GPU at
0000:83:00.0 has fallen off the bus.

thanks for any help inadvance

Perhaps you need better cooling system. GeForce cards have problems with sustained heavy loads.

Yes, that does appears to be the problem as we have a server with horizontal mobo, with 4 580s mounted next to each other vertically. and it gets up to 100C under full load. Any suggestions on how to cool it?