Hello, I’m training RNN Model on Tesla P40.
When I ran the python script, I got CUDA error: an illegal memory access was encountered.
My test environment is
OS : CentOS7
Graphic card : Tesla P40
nvidia driver version : 440.33.01
CUDA version : 10.2
cudnn : 7605
Pytorch : 1.5.0
I tested many options below. But illegal access error happened.
- from torch.backends import cudnn
- torch.backends.cudnn.enabled = False
As I checked nvidia-smi dmon, Memory was not allocated.
I think the problem is memory allocation. But there is no clue.
How can I improve this situation?
Please, give me any solution.