Hi, we have a problem with one of our GTX 2080Ti that we use for deep learning. We have 2 of the aforementioned GPUs and one of them works just fine, however, the other one, explodes the activations of the convolutional layers to inf, resulting nans in the losses and gradients.
Here is the PyTorch forums thread, where we are trying to figure out the problem (other deep learning practitioners have the same issues): https://discuss.pytorch.org/t/different-losses-on-2-different-machines/36446
GPU: RTX 2080Ti
Any help would be appreciated, thank you