CUDA Error: CUDA driver version is insufficient for CUDA runtime version on Windows 2016 Server after reboot

After installing Cuda 9.1 everything is working fine. I then reboot the machine and start getting:

CUDA Error: CUDA driver version is insufficient for CUDA runtime version

I then reinstall Cuda 9.1 and everything works again. But then after another reboot the error above. What’s going on here?

I’m using cuda_9.1.85._windows. Tried installing all the patches but it doesn’t make any difference. I basically need to reinstall cuda everytime I want to use it :(

When you reboot and have the error, what is the GPU driver version?

What happens if you run nvidia-smi from a command prompt in the failing state?

In the failing state:

±----------------------------------------------------------------------------+
| NVIDIA-SMI 386.07 Driver Version: 386.07 |
|-------------------------------±---------------------±---------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 TCC | 0000F9BF:00:00.0 Off | 0 |
| N/A 30C P8 32W / 149W | 0MiB / 11444MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

After re-installing cuda_9.1.85_win10 again:

±----------------------------------------------------------------------------+
| NVIDIA-SMI 388.19 Driver Version: 388.19 |
|-------------------------------±---------------------±---------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 TCC | 0000F9BF:00:00.0 Off | 0 |
| N/A 30C P8 33W / 149W | 0MiB / 11444MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

And the problem goes away :)