We purchased some machines with 6/8 GPUs installed. These machines suffer from random freezes since then. If you start running some job using the GPU, especially when the job begins or ends, the machines are likely to freeze. We tried a number of configurations (see below), but doesn’t resolve the issue.
Linux: Ubuntu 14.04, Ubuntu 16.04
Kernel: 3.13.0-68, 4.4.0-66
GPUs: Pascal Titan X, 1080
Nvidia drivers: 375.20, 375.39, 378.13, among a number of others
When we generated the nvidia log, it led to a hang, too.