I am not able to train my model to completion. I get the following error:
2020-09-01 17:05:07.819801: E tensorflow/stream_executor/dnn.cc:616] CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/stream_executor/cuda/cuda_dnn.cc(1892): 'cudnnRNNForwardTraining( cudnn.handle(), rnn_desc.handle(),
model_dims.max_seq_length, input_desc.handles(), input_data.opaque(), input_h_desc.handle(), input_h_data.opaque(),
input_c_desc.handle(), input_c_data.opaque(), rnn_desc.params_handle(), params.opaque(), output_desc.handles(), output_data-
>opaque(), output_h_desc.handle(), output_h_data->opaque(), output_c_desc.handle(), output_c_data->opaque(),
workspace.opaque(), workspace.size(), reserve_space.opaque(), reserve_space.size())'
2020-09-01 17:05:07.819965: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query
event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure
2020-09-01 17:05:07.861316: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at cudnn_rnn_ops.cc:1517 :
Internal: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 ,
[num_layers, input_size, num_units, dir_count, max_seq_length, batch_size, cell_num_units]: [1, 300, 150, 1, 20, 620, 150]
2020-09-01 17:05:07.873534: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:220] Unexpected Event status: 1
The weird thing is that when my room is cool, 70°F, I don’t get this error. Only when the temperature of my room is 75°F and above!