System info:
GPU Type: Tesla T4
Nvidia Driver Version: 418.87.01
CUDA Version: 10.1.243
CUDNN Version: 7.6.3
Python Version (if applicable): 3.7.4
TensorFlow Version (if applicable):1.14.0
Operating System + Version: Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64)
Hi, I am training tensorflow version Tacotron2 model with mix precision training.
after some training iterations , when doing validation, an error occurs, the detailed error info is like below:
Exiting due to exception: 2 root error(s) found. (0) Invalid argument: TensorArray dtype is float but Op is trying to write dtype half. [[node Tacotron_model/inference/encoder_LSTM/bidirectional_rnn/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (defined at /home/yichao.li/lite-tacotron2/tacotron/models/modules.py:225) ]] [[strided_slice_51/_7343]] (1) Invalid argument: TensorArray dtype is float but Op is trying to write dtype half. [[node Tacotron_model/inference/encoder_LSTM/bidirectional_rnn/fw/fw/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (defined at /home/yichao.li/lite-tacotron2/tacotron/models/modules.py:225) ]] 0 successful operations. 0 derived errors ignored.
Do I need to turn off auto mixed precision on evaluation time?
could you help to clarify this? how to fix?