Illegal access memory error

Hello, I’m training RNN Model on Tesla P40.
When I ran the python script, I got CUDA error: an illegal memory access was encountered.

My test environment is
OS : CentOS7
Graphic card : Tesla P40
nvidia driver version : 440.33.01
CUDA version : 10.2
cudnn : 7605
Pytorch : 1.5.0

I tested many options below. But illegal access error happened.

  1. CUDA_LAUNCH_BLOCKING=1
  2. from torch.backends import cudnn
    torch.backends.cudnn.benchmark=True
  3. torch.backends.cudnn.enabled = False
  4. torch.cuda.set_device(0)

As I checked nvidia-smi dmon, Memory was not allocated.
I think the problem is memory allocation. But there is no clue.

How can I improve this situation?
Please, give me any solution.

Thank you.