Hello, I’m training RNN Model on Tesla P40.
When I ran the python script, I got CUDA error: an illegal memory access was encountered.
My test environment is
OS : CentOS7
Graphic card : Tesla P40
nvidia driver version : 440.33.01
CUDA version : 10.2
cudnn : 7605
Pytorch : 1.5.0
I tested many options below. But illegal access error happened.
- CUDA_LAUNCH_BLOCKING=1
- from torch.backends import cudnn
torch.backends.cudnn.benchmark=True - torch.backends.cudnn.enabled = False
- torch.cuda.set_device(0)
As I checked nvidia-smi dmon, Memory was not allocated.
I think the problem is memory allocation. But there is no clue.
How can I improve this situation?
Please, give me any solution.
Thank you.