tensorflow CUDA_ERROR_MIS ALIGNED_ADDRESS: misaligned address

envs:
Ubuntu18.04
cuda 10
cudnn7
tensorflow-gpu 1.13.1

When I train through TensorFlow, I get the following error:
2020-03-01 21:54:42.824973: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2020-03-01 21:54:44.478783: E tensorflow/stream_executor/cuda/cuda_driver.cc:981] failed to synchronize the stop event: CUDA_ERROR_MI$
ALIGNED_ADDRESS: misaligned address
2020-03-01 21:54:44.478840: E tensorflow/stream_executor/cuda/cuda_timer.cc:55] Internal: error destroying CUDA event in context 0x55$
da6259f30: CUDA_ERROR_MISALIGNED_ADDRESS: misaligned address
2020-03-01 21:54:44.478853: E tensorflow/stream_executor/cuda/cuda_timer.cc:60] Internal: error destroying CUDA event in context 0x55$
da6259f30: CUDA_ERROR_MISALIGNED_ADDRESS: misaligned address
2020-03-01 21:54:44.478891: F tensorflow/stream_executor/cuda/cuda_dnn.cc:194] Check failed: status == CUDNN_STATUS_SUCCESS (7 vs. 0)$
ailed to set cuDNN stream.
已放弃 (核心已转储)

What version of cudnn 7 are you using? If not the latest, can you update and try again? https://developer.nvidia.com/rdp/cudnn-download

If you still see this failure, is it possible for you to provide a small reproducer?

Yes, I use cudnn-10.0-linux-x64-v7.6.5.32, cuda10.0.130_410_48, NVIDIA-Linux-x86_64-418.88. pip install tensorflow, tf is the latest version.
I reinstalled anaconda, and I also tried cuda10 + cudnn7.5.1.10 and pytorch 1.4, 1.3, 1.2, 1.1 almost the same error. Now I don’t know how I can do it. Maybe I reinstalled Ubuntu?
Or is it actually a hardware problem? I was on a training mission a month ago, and then he suddenly reported an error, and then he couldn’t use it.
It’s really tiring!

What mean reproducer?What should I do?

A reproducer is a simple script that we can use to generate and debug the error internally. Ideally it uses generated input data, and includes only the fragment of your original model needed to trigger the error.

I have never used it when using python

“script” is meant generically. For example, a small C++ program would be fine too.

If a reproducer cannot be provided, you’ll need to debug on your end. The error message above suggests that a GPU kernel is generating an exception. You can try narrowing down which kernel is doing this by enabling debugging output in TF (export TF_CPP_MIN_VLOG_LEVEL=2) and configuring cuda to execute GPU code synchronously (export CUDA_LAUNCH_BLOCKING=1). If you are familiar with debuggers like GDB, you might also want to check out cuda-gdb https://docs.nvidia.com/cuda/cuda-gdb/index.html.