Activations and Gradients blow up on one of RTX2080Ti

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

Hi, we have a problem with one of our GTX 2080Ti that we use for deep learning. We have 2 of the aforementioned GPUs and one of them works just fine, however, the other one, explodes the activations of the convolutional layers to inf, resulting nans in the losses and gradients.

Here is the PyTorch forums thread, where we are trying to figure out the problem (other deep learning practitioners have the same issues): Different Losses on 2 different machines - autograd - PyTorch Forums

cuDNN: v7.4.2
CUDA: 10.0.130
PyTorch: 1.0.1
GPU: RTX 2080Ti

Any help would be appreciated, thank you