I have been using tensorflow nvidia-dockers since the 18.09 version and didn’t face any error. Recently, I am facing the following error all of a sudden:
“tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version”
I believe the drivers are installed with the docker image itself. So, I don’t see why there is a compatibility issue. I have also checked the CUDA_DRIVER_VERSION, CUDA_VERSION etc environment variables and they satisfy requirements mentioned in the documentation.
When I stop the container and launch again, it is working sometimes. Does anyone have the same problem ? Please help me out with this.