When I put in used K80s, I needed to ensure I have an updated BIOS and enable 4G addressing (and yes, of course, cooling). Is there anything unexpected for the P100? If it works for the K80, will that be enough for a P100?
Should there be any problems with three generations in the same box? GTX 980, K80, P100? If I have a good experience with the P100, I’m willing to replace the K80s with P100s (Pascal instead of Kepler, and the K80 is already deprecated in CUDA), but unless I know it will work with P100s, I won’t buy all of them.
Mixing of GPU generations should not be a problem as long as all generations can be covered by the same driver and the same CUDA version. Example: Using both sm_80 and sm_30 GPUs in the same system is not going to work, best I know. The most disparate mixes of GPUs I have tried myself used GPUs two generations apart, but the older generation GPUs in those cases were only used for the GUI display, not for compute.
I can’t really speak to the K80 to P100 transition due to lack of hands-on experience. I am under the impression that SBIOS requirements for Tesla-brand GPUs are quite similar.