Repeated Beeping Noise and Loss Rapidly Decreasing When Training on Keras+Tensorflow

AI & Data Science Deep Learning (Training & Inference)
1

CUDA: 10
Cudnn: 7.3.1
python: 3.6
Os: Windows 10
GPU: RTX 2080

I installed tensorflow-gpu from source back in november and things seemed to be working. Recently I trained an LSTM using Keras and Tensorflow which worked fine until I tried to transfer the weights to a new model, with another layer, and continue training. Now whenever I try training a model there are 3 cases that happen:

1. I get the error ‘InternalError: Blas GEMM launch failed.’
Restarting the notebook kernal seems to fix it.

2. GPU Sync failed
Restarting pc sorta fixes it

3. Training runs, but computer makes quick repeated beeping sounds the entire time and the loss quickly decreases towards 0. It happens every single time.

Note: The beepings only occur during training and never in games or other gpu stuff. I first assumed it was a power issue, but now it seems to be something wrong with the tensorflow install.

What I’ve tried: Updating my drivers, re-seating gpu and ensuring connections are there and windows and tensorflow see it, reinstalling tensorflow with different versions of cudnn (7.3.0, 7.4.1) and python 3.7, always cuda 10.

Anyone know what could be causing this issue?
Thanks for any help! Let me know if you need any more information.

2

Update:

1. & 2. TensorFlow: InternalError: Blas SGEMM launch failed - Stack Overflow
Following these steps seems to have fixed the first 2 issues.
Note: I also switched to the Cudnn optimized models.

3. I noticed a very faint buzzing noise while playing Shadow of the Tomb Raider, and so I’m now realizing the noise is coil whine, as it changes frequency with the GPU load or the training batch size. The loss problem seems to be due to vanishing gradients in the network, and modifying the optimizer params (type and learning rate) fixed it.

https://stackoverflow.com/questions/46224598/python-keras-lstm-learning-converges-too-fast-on-high-loss?fbclid=IwAR0-6ikl5p460UUeOeVKpL84WRmYkGPwUltpeckMRkpF89MA65drYgLbK3I