Different performance from different GPUs with Identical Code

I have some experience using CUDA with multiple GPUs (I’m running 14 in total, distributed across several machines). Of the 14 cards I originally ordered (GTX 570, from EVGA), 4 tested positive for faulty memory. I used memtestG80 for this. These cards with bad memory were also yielding erroneous results in computations. I recommend checking the memory if you can.