CUDA sample throwing error

When I try to run the BlackScholes sample I get this error

`CUDA error at ../../common/inc/helper_cuda.h:777 code=35(cudaErrorInsufficientDriver) "cudaGetDeviceCount(&device_count)"`

In my WSL2 distro CLI when I type

uname -r

it shows

`4.19.121-microsoft-standard`

However on windows powershell the command

wsl cat /proc/version

shows

`Linux version 4.4.0-20170-Microsoft (Microsoft@Microsoft.com) (gcc version 5.4.0 (GCC) ) #1000-Microsoft Thu Jul 10 14:19:00 PST 2020`

Shouldn’t it be showing the same version i.e. 4.19.121 ?

I suspect this might be the cause of the error I get when running CUDA sample.

Thanks

3 Likes

Hello and thanks for reaching out.

The mismatching is version is likely caused by the fact that you have multiple WSL setup on your system. And when typing “wsl cat /proc/version” the default one will be used to run the command which might be different from the one you used to run WSL2. You can check that by running wsl -l and see which one is used by default.

For the CUDA issues verify the following:

  • You have installed the prototype driver that has WSL support: https://developer.nvidia.com/cuda/wsl/download
  • When you installed the CUDA Toolkit check that you have not installed the native driver as well (that driver will shadow the WSL driver and will be picked instead). This is an easy mistake to make if you used the .deb file (the CUDA one will install the native driver as well). You will find couple of forum post on that topic (and various way to get back to a good state).
  • If you still have an issue after that could you post here a dxdiag log (run dxdiag on your windows machine and you will find a button on the GUI to save the logs) and we can look at it.

Thanks !

Hi Thanks for the answer, I have uninstalled Wsl 1 distro and now the wsl command shows the correct version as well.

But I still have the CUDA issue. I followed exactly the steps mentioned on nvidia site to just install the toolkit. I am not sure if the linux drivers got installed with it? How can I check this?

I have attached dxdiag output DxDiag.txt (117.1 KB)

Thanks

Hello - I am having this same issue with my WSL/CUDA installation. Can you please link to forum posts discussing the native driver?

I have done the instruction
CUDA on WSL :: CUDA Toolkit Documentation, having updated kernel:

uname -r

4.19.121-microsoft-standard

and after successfullly installed

apt-get install -y cuda-toolkit-11-0

Reading package lists… Done
Building dependency tree
Reading state information… Done
cuda-toolkit-11-0 is already the newest version (11.0.3-1).
The following packages were automatically installed and are no longer required:
bridge-utils dns-root-data dnsmasq-base libidn11 ubuntu-fan
Use ‘apt autoremove’ to remove them.
0 upgraded, 0 newly installed, 0 to remove and 37 not upgraded.

I have the same problem.

./BlackScholes

[./BlackScholes] - Starting…
CUDA error at …/…/common/inc/helper_cuda.h:777 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”

7 Likes

CUDA error at …/…/common/inc/helper_cuda.h:777 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”

I have the same problem. Anyone has found a solution yet? Thank you.

Same issue, anyone found a solution yet?

Hello,

Assuming you are on WSL2 (we don’t support WSL1) this is usually caused by the wrong display driver being installed. This usually happens if instead of just installing the toolkit package a packages that has a dependency on the native linux driver was installed as well.

You should be able to purge that from your package manager (sudo apt --purge followed by the nvidia-driver that got installed) but the process might be a bit tricky.

A quick way to verify that this is what is happening is by checking which CUDA driver is getting used:

  • The WSL drivers are in /usr/lib/wsl/…
  • Run strace on your binary and look in the logs for libcuda, if it comes from a different place than /usr/lib/wsl/… (or one of the subfolders) the wrong driver is being used.

Hope this would help,

1 Like

I had the same issue and it was resolved by updating to the latest insider build. It is not enough to have the later WSL2 version. You also need the insider build and beta is not enough.

Encountering same problem, removed WSL 1 using the following command:
wslconfig.exe /unregister Legacy

(see: User questions - migrating from legacy WSL and interface with MOBAxterm · Issue #3022 · microsoft/WSL · GitHub)

It did not help.

My issue is likely that I am on Windows Version 19041.508 and not 20145 or later like the guide suggests. I guess I have to either wait (I think I will) or run a scary beta version of Windows which I am not ready to do.

image

image

4 Likes

Thanks. I followed your instructions and libcuda seems to be in the right place:

openat(AT_FDCWD, “/usr/lib/wsl/lib/libcuda.so.1”, O_RDONLY|O_CLOEXEC) = 3

Still, the same error persists. What seems to be failing is the file dev/dxg:

openat(AT_FDCWD, “/dev/dxg”, O_RDONLY) = -1 ENOENT (No such file or directory)
munmap(0x7f1a6d3a3000, 44390) = 0
futex(0x7f1a6ce2afc0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x563f8cf39a90, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, “CUDA error at …/…/common/inc/h”…, 123CUDA error at …/…/common/inc/helper_cuda.h:777 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”
) = 123
write(1, “[./BlackScholes] - Starting…\n”, 31) = 31
exit_group(1) = ?
+++ exited with 1 +++

Any ideas how to fix this? Thank you

@iribo what’s your Windows build (winver.exe)? Can you post the output of uname -a from bash?

@onomatopellan Output of uname -a from WSL2 bash:

Linux DESKTOP-K9VH9QQ 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

My Windows 10 build is: Version 2004 (OS Build 20215.1000).

@iribo Thanks. Your kernel 4.19.104 is too old. You need to update to at least 4.19.121 running wsl.exe --update from an elevated cmd.

Further instructions in CUDA on WSL :: CUDA Toolkit Documentation

3 Likes

@onomatopellan Great! That solved the issue. The BlackScholes sample is now executing correctly. Thank you very much for your help.

2 Likes

@rboissel can you give more detailed instructions about how to purge old driver version and get back to the one from WSL2?

I use windows 1909 (18363.1049)
WSL2
windows nvidia drivers installed from https://developer.nvidia.com/cuda/wsl/download (quadro 460.15)

but unfortunately when installing cuda, I run sudo apt-get -y install cuda and I should have run sudo apt-get install -y cuda-toolkit-11-0? (I followed these instructions: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=deblocal)

I have the same error:
[/usr/local/cuda/samples/4_Finance/BlackScholes/BlackScholes] - Starting…
CUDA error at …/…/common/inc/helper_cuda.h:777 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”

I guess it is using an old driver version and not the proper one.

For CUDA in WSL2 you need to be in a Windows Insider build from Dev Channel. (20145 or higher)

1 Like

sudo wsl.exe --update from powershell with admin privileges did if for me.

For all the guys meeting this error
CUDA error at …/…/common/inc/helper_cuda.h:777 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”

Here is my solution

  1. check whether you have followed the instructions in Doc strictly.
    If you have installed WSL distro before, most of the time you cannot update it to WSL2 directly (I tried many times all fail).
    What you need to do is uninstalling the WSL distro and installing the WSL2 distro.
    For example, I have installed Ubuntu20.04 one month ago and it was WSL. And now what I am going to do is uninstalling that distro, install WSL2 from the Microsoft website (download the package and install directly), and then install Ubuntu-20.04/18.04/…
    You can use wsl -l -v in your PowerShell to check whether your distro is using WSL2

    !!! If necessary backup your important files, backup what you have installed in the previous distro. (I make installed packages and software into a txt file(names))!!!

  2. Windows insider part, and make it dev mode.
    Updating may take a long time. From my side, it took about 20 mins.
    If you do not have /usr/lib/wsl folder in your WSL2 Ubuntu, it means you fail in installing the windows insider step.(I do not have this folder before installing Windows insider successfully)

  3. if you succeeded in the steps above, do [sudo make clean] under /usr/local/lib/cuda/sample/4_finance/black-whateversomething and recompile the file (sudo make).

3 Likes