deviceQuery fail after CUDA 10.1 installation

[chengchen330]

Hi all,

I have been trying to install CUDA 10.1 on my Windows 10 Dell Workstation which has a Geforce RTX 2080 card. I had the lastest version of driver installed. The installation of CUDA went smoothly with no problem, but when I ran deviceQuery.exe, it could not detect the graphic card. Please see the messages from the Windows Command Prompt:

%%%%%%

C: \ProgramData\NVIDA Corporation\CUDA Samples\v10.1\bin\win64\Release>deviceQuery
deviceQuery Starting…
CUDA Device Query (Runtime API) version (CUDART static linking)
C: \ProgramData\NVIDA Corporation\CUDA Samples\v10.1\bin\win64\Release>

%%%%%%

Does anyone has experience with this problem and could give some advices?

Many thanks!

[chengchen330]

I should add that when I ran nvidia-smi.exe, it could detect the driver version 441.28, and CUDA version 10.2, though what I installed was really 10.1!

[mnicely]

I’m running Windows 10 w/ Quadro M1200 w/ R441.22.
With CUDA 10.1

...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.2, CUDA Runtime Version = 10.1, NumDevs = 1
Result = PASS

With CUDA 10.2

...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.2, CUDA Runtime Version = 10.2, NumDevs = 1
Result = PASS

Notice that nvidia-smi displays the CUDA Driver Version. It’s telling you that your CUDA Driver supports CUDA Runtime up to 10.2.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 441.22       Driver Version: 441.22       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro M1200       WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   38C    P8    N/A /  N/A |     40MiB /  4096MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

[chengchen330]

Thanks for the message! It’s good to know that Nvidia-smi displays the the version it support. But do you have any idea that why when I ran deviceQuery, CUDA could not detect my graphic card please?

[mnicely]

In my experience, if deviceQuery fails, nvidia-smi would also fail. That’s not what you’re experiencing. I suggest removing and reinstalling CUDA and drivers.

[chengchen330]

Thanks! I’ve tried to install CUDA 9.0, but has the same problem. Do you think it worth to try older version of drivers?

[mnicely]

Yes. With issues such as this one, confirm the issue occurs with the driver installed with the CUDA toolkit. Remove your drivers and install 10.2 and let me know if you still have issues.

[chengchen330]

Not sure I understand what you mean.I can’t install 10.2 because the whole purpose of setting up CUDA is for my Matlab to do GPU-based computation. And the latest version of Matlab 2019b only supports CUDA up to 10.1 (GPU Computing Requirements - MATLAB & Simulink - MathWorks United Kingdom).

And I installed the graphic card driver using GeForceExperience which automatically detected the last version of driver.

[mnicely]

I only suggested 10.2 because it’s the latest. Any version will do, as long you confirm the issue occurs with the driver in the CUDA toolkit.

As an example, 10.1 comes with R418.96. While the latest Windows driver is R441.66. Make sure deviceQuery works with R418.96, then upgrade to R441.66 and retest.

I have confirmed deviceQuery and nvidia-smi work on my machine with R441.66 as well.

Also, I don’t think MathWorks is saying Matlab 2019b only works with 10.1. More likely, to use all the available features in 2019b you need 10.1 or newer.

[chengchen330]

I just received the reply from Matlab Support, saying "MATLAB R2019b will only support CUDA Toolkit 10.1 or a previous version of the CUDA Toolkit. I recommend that that you install CUDA Toolkit 10.1
" So, I have to install 10.1 or lower.

Since I have done uninstall-reinstall many time, can I confirm how to do a CLEAN Uninstallation of the graphic driver and CUDA? I did it through Control Panel-Programs-Uninstall or Change a Program. Is this the right way to uninstall? Do I have to shut down the power of computer and then restart to make sure the uninstall/installation works properly?

Many thanks for your help!

[mnicely]

I don’t use Windows often but the CUDA Toolkit documentation says to uninstall everything through the Windows Control Panel. I suggest you do the same thing for the Windows driver and Geforce stuff.

https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#uninstall-cuda-software

Reboot. Confirm no NVIDIA driver is installed. Reinstall CUDA. Rerun deviceQuery.

[chengchen330]

I did uninstall all components of CUDA 10.1, including Gefore Experience and the Graphic Card driver 441.41. I also rebooted the computer. When I double check the program list in Control Panel-Programs-Uninstall or Change a Program, there is no Graphic Card Driver or CUDA. But, I found I could still open NVIDIA Control Panel, it showed I have a driver which is 441.41. And, I could still run nvidia-smi.exe which also detect the driver 441.41 and CUDA. Please see the screenshot I enclose. I feel there is something wrong here. It seems that GPU driver and CUDA could not be cleanly removed??? Or I didn’t use the right method???

[chengchen330]

Hi all, here are some updates:

1. I went to Device Manager to uninstall the driver 441.41. And I uninstall NIVIDA control Panel. Restart. This time I can confirm that driver 441.41 is successfully removed. And about 5mins after I restarted, Windows automatically installed for me the driver 432.00.

2. Now I started install CUDA 10.1, it’s cuda_10.1.105_418.96_win10.exe from: CUDA Toolkit 10.1 original Archive | NVIDIA Developer. I chose the Express install. During the installation, it showed that it wiped out my previous driver(432.00) to install the driver 418.96.

3. Immediately following the installation of CUDA 10.1, I restarted the computer. I checked Device Manager and Nvidia-smi, and found there are actually problem with the driver 418.96. Please see enclosed screenshots.

[chengchen330]

So, I can’t follow mnicely’s advice to check whether CUDA 10.1 works with driver 418.96 on my computer. Then I go to GeforceExperience which was just installed with CUDA 10.1, to my surprise, it said, there is no driver in the system! Please see the enclosed screenshot.

[chengchen330]

I then have to upgrade my driver using GeforceExperience to the lastest 441.66. Restart. And now there is no problem about the display driver in device manager as I saw after the installation of 418.96 with CUDA 10.1. Run nvidia-smi, I can see the driver 441.66. HOWEVER, after compile deviceQuery.sln using Visual Studio 2013, and ran deviceQuery, I still got the same error as I have when I started this thread…

%%%%%%

C: \ProgramData\NVIDA Corporation\CUDA Samples\v10.1\bin\win64\Release>deviceQuery
deviceQuery Starting…
CUDA Device Query (Runtime API) version (CUDART static linking)
C: \ProgramData\NVIDA Corporation\CUDA Samples\v10.1\bin\win64\Release>

%%%%%%

And I opened Matlab2019b, ran gpuDevice, Matlab crashed again…

Anyone have any idea about what might cause the problem. And do you think there is something weird about my Windows 10 system and I should consider reshape my OS to Linux???

[Robert_Crovella]

I really don’t know what the problem is. However my first guess would be a corrupted CUDA install.

Do you have a requirement for a particular CUDA version? For example do you need to use CUDA 10.1?

[chengchen330]

I have to install CUDA 10.1 or lower. But previously I tried CUDA 9.0, Matlab still crashes…

[Robert_Crovella]

Ok, can you try installing the latest CUDA 10.1 (Update 2) release

https://developer.nvidia.com/cuda-10.1-download-archive-update2?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exelocal

When you run the installer, deselect the option to install the bundled driver. That should leave your 441.66 driver intact, which seems to be working correctly.

Make sure you have a properly installed version of Visual Studio installed before running the above steps.

https://docs.nvidia.com/cuda/archive/10.1/cuda-installation-guide-microsoft-windows/index.html

(2013 should be fine, as long as it is not the community edition)

Then build the deviceQuery project in VS2013, and attempt to run it.

I assume you are working locally on this machine?

Let’s try and get the deviceQuery running before tackling matlab.

[chengchen330]

Hi Robert, thanks a lot for these suggestions! To clarify and make sure I really understand, there are some questions:

1. Do I need to first uninstall the current CUDA 10.1 with all its components and GeforceExperience, et al…, but leave the driver 441.66 intact, before installing the latest CUDA 10.1 (Update2)?

2. I don’t quite remember exactly which version of VS2013 I installed. It’s one with update 5. I will double check tomorrow to confirm. It’s likely I installed the first one on this page (Visual Studio Community 2013 with Update 5, Azure DevOps Services | Sign In). Do you mean I should actually install the 2nd one (Visual Studio 2013 Update 5)?

3. What do you mean by saying “working locally on this machine”?

[Robert_Crovella]

1. It’s not necessary to uninstall previous CUDA installs. It is necessary to make sure that when you build deviceQuery, you use the code and libraries that were installed with the latest install I suggested. That should be just a matter of navigating to the proper sample code project file.

2. If you’re using Visual Studio community, I would stop doing that and install a “full” version of Visual Studio. This may cost money depending on which version, etc. You’ll need to learn about Visual Studio and all it’s myriad variety yourself, I won’t be able to give you a tutorial on it. For example, I can’t access the link you provided. Microsoft puts login walls in front of it, I guess.

3. By locally, I mean you are sitting in front of the machine looking at the display that is directly cabled to the graphics card you are using. This is distinguished from connecting remotely to the box over RDP or some other remoting solution.