Os: Windows 10
GPU: RTX 2080
I installed tensorflow-gpu from source back in november and things seemed to be working. Recently I trained an LSTM using Keras and Tensorflow which worked fine until I tried to transfer the weights to a new model, with another layer, and continue training. Now whenever I try training a model there are 3 cases that happen:
1. I get the error ‘InternalError: Blas GEMM launch failed.’
Restarting the notebook kernal seems to fix it.
2. GPU Sync failed
Restarting pc sorta fixes it
3. Training runs, but computer makes quick repeated beeping sounds the entire time and the loss quickly decreases towards 0. It happens every single time.
Note: The beepings only occur during training and never in games or other gpu stuff. I first assumed it was a power issue, but now it seems to be something wrong with the tensorflow install.
What I’ve tried: Updating my drivers, re-seating gpu and ensuring connections are there and windows and tensorflow see it, reinstalling tensorflow with different versions of cudnn (7.3.0, 7.4.1) and python 3.7, always cuda 10.
Anyone know what could be causing this issue?
Thanks for any help! Let me know if you need any more information.