CUDA sample throwing error

@rboissel can you give more detailed instructions about how to purge old driver version and get back to the one from WSL2?

I use windows 1909 (18363.1049)
WSL2
windows nvidia drivers installed from https://developer.nvidia.com/cuda/wsl/download (quadro 460.15)

but unfortunately when installing cuda, I run sudo apt-get -y install cuda and I should have run sudo apt-get install -y cuda-toolkit-11-0? (I followed these instructions: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=deblocal)

I have the same error:
[/usr/local/cuda/samples/4_Finance/BlackScholes/BlackScholes] - Starting…
CUDA error at …/…/common/inc/helper_cuda.h:777 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”

I guess it is using an old driver version and not the proper one.

For CUDA in WSL2 you need to be in a Windows Insider build from Dev Channel. (20145 or higher)

sudo wsl.exe --update from powershell with admin privileges did if for me.

For all the guys meeting this error
CUDA error at …/…/common/inc/helper_cuda.h:777 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”

Here is my solution

  1. check whether you have followed the instructions in Doc strictly.
    If you have installed WSL distro before, most of the time you cannot update it to WSL2 directly (I tried many times all fail).
    What you need to do is uninstalling the WSL distro and installing the WSL2 distro.
    For example, I have installed Ubuntu20.04 one month ago and it was WSL. And now what I am going to do is uninstalling that distro, install WSL2 from the Microsoft website (download the package and install directly), and then install Ubuntu-20.04/18.04/…
    You can use wsl -l -v in your PowerShell to check whether your distro is using WSL2

    !!! If necessary backup your important files, backup what you have installed in the previous distro. (I make installed packages and software into a txt file(names))!!!

  2. Windows insider part, and make it dev mode.
    Updating may take a long time. From my side, it took about 20 mins.
    If you do not have /usr/lib/wsl folder in your WSL2 Ubuntu, it means you fail in installing the windows insider step.(I do not have this folder before installing Windows insider successfully)

  3. if you succeeded in the steps above, do [sudo make clean] under /usr/local/lib/cuda/sample/4_finance/black-whateversomething and recompile the file (sudo make).

3 Likes

A solution that worked for me Try:

  1. Opening a Windows Start Menu
  2. Type “Run” then enter
  3. Type “control /name Microsoft.DeviceManager” then enter
  4. Under “Device adapters” verify there’s no warnings or issues with Nvidia device
  5. If Windows states there’s a problem with device, follow simplified instructions provided by Microsoft at resource: https://support.microsoft.com/en-us/windows/fix-graphics-device-problems-with-error-code-43-6f6ae1ec-0bbe-a848-142e-0c6190502842

I had the same problem and was banging my head for half a day. My problem was that I installed the WSL compatible driver (https://developer.nvidia.com/cuda/wsl/download ) before switching to new Windows Build which supports CUDA on WSL 2. I found for instance that libcuda.so was missing from C:\Windows\System32\lxss\lib. I had to re-install the wsl driver and after re-installing the driver everything worked.

I’m seeing the same problem and I’ve been through this entire post and tried everything. Here’s what I have done:

I have an Alienware M17 with GeForce RTX 2080 Super
Downloaded NVIDIA drivers for WSL (https://developer.nvidia.com/cuda/wsl/download)
Subscribed to Windows Insider Dev and applied the updates
Activated WSL 2 to give me Linux version 4.19.128-microsoft-standard
Verified that I have /usr/lib/wsl (to prove i have insider dev active)
Dowloaded CUDA 11.1 (Linux WSL-Ubuntu 2.0 x86_64) via https://developer.nvidia.com/cuda-downloads using deb(network) version.
Installed Cuda 11.1 via the recommended steps on that page
Checked and I do have files in /usr/local/cuda-11.1 to compile against.
Ensured that the Makefile in /usr/local/cuda-11.1/samples/0_Simple/clock use ‘CUDA_PATH ?= /usr/local/cuda-11.1’
Checked that the files /usr/lib/wsl/lib/libcuda.so.1 etc. exist
Ran strace on the ‘clock’ program and verified that it is loading ‘/usr/lib/wsl/lib/libcuda.so.1’
After all this I still get the error ‘CUDA error at …/…/common/inc/helper_cuda.h:779 code=35(cudaErrorInsufficientDriver) “cudaGetDeviceCount(&device_count)”’

I’m running out of ideas. Can anyone please help?

Hello everyone. I’m facing the same problem, but despite having followed all the guidelines in the post, nothing went right. Here is my procedure:

I have a Dell Inspiron 15 7000 Gaming notebook, with an NVIDIA GeForce GTX 1050 Ti card, and Windows 10 Pro Insider Preview, 2004, 21277.1000, with Windows Update Advanced options enabled to enable recommended Microsoft updates and run the check again . Several times I followed the tutorial step ‘https://docs.nvidia.com/cuda/wsl-user-guide/index.html#running-cuda’ for installing and configuring the CUDA Toolkit:


Installing NVIDIA Drivers on Windows:
460.89 - 12/15/2020


Installing WSL 2: I had already installed it in advance through the tutorial ‘https://docs.microsoft.com/pt-br/windows/wsl/install-win10’ - 9/15/2020


PowerShell (admin):

Linux version:
PS C:\Users\sanam> wsl cat /proc/version
Linux version 5.4.72-microsoft-standard-WSL2 (oe-user @ oe-host) (gcc version 8.2.0 (GCC)) # 1 SMP Wed Oct 28 23:40:43 UTC 2020

Ubuntu-18.04 installed by Microsoft Store:
PS C:\Users\sanam> wsl --list --verbose
NAME STATE VERSION
Ubuntu-18.04 Running 2

GPU recognition:
PS C:\Users\sanam> nvidia-smi

Activating Ubuntu:
PS C:\Users\sanam> wsl


Ubuntu CLI:

/mnt/c/Users/sanam$ cd ~
~$ sudo su

~# sudo apt update
~# sudo apt upgrade

Verify You Have a CUDA-Capable GPU (‘https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html’):
~# update-pciids
Downloaded daily snapshot dated 2020-12-13 03:15:01
~# lspci | grep -i nvidia
“Nothing happens!”

Verify You Have a Supported Version of Linux:
~# uname -m
x86_64
~# cat /etc/*release

PRETTY_NAME = “Ubuntu 18.04.5 LTS”

Verify the System Has gcc Installed:
~# gcc --version
Command ‘gcc’ not found, but can be installed with:
apt install gcc
~# apt update
~# gcc --version
Copyright © 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSEgcc (Ubuntu 7.5.0-3ubuntu1 ~ 18.04) 7.5.0

Verify the System has the Correct Kernel Headers and Development Packages Installed:
~# uname -r
5.4.72-microsoft-standard-WSL2

Setting up CUDA Toolkit:
~# apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
~# sh -c ‘echo “deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /”> /etc/apt/sources.list.d/cuda.list’
~# apt-get update
~# apt-get install -y cuda-toolkit-11-0

Running CUDA Applications:
~# cd /usr/local/cuda/samples/4_Finance/BlackScholes
~# make
~# ./BlackScholes
[./BlackScholes] - Starting …
CUDA error at …/…/common/inc/helper_cuda.h:777 code = 35 (cudaErrorInsufficientDriver) “cudaGetDeviceCount (& device_count)”

Since then I am looking for how to solve this problem. If possible, I would appreciate your help.

Thanks.

2 Likes

I encountered the exact same problem after update to CUDA 11.2 and 465.21 driver, I have only one WSL2 setup. I think it’s a NV bug. please help

3 Likes

Yeah dude I cannot fathom this shit dude , stuck since Sep-Nov

I’ve been doing in Beta channel. Thanks for the solution. I’ll try it again!

same problem here. I tried everything as said in manual, but nothing works out. NVIDIA team please solve.

Board: Jetson AGX XAVIER
Jetpack: 4.4.1

Dear, this time everything went well!
After the new update of Windows 10 Insider Preview 21286.1000 (rs_prerelease) and as well as the NVIDA driver to 465.21, I decided to try again, performing the following steps:
Note: You must already be enrolled in the Windows Insider Program.

  1. Uninstall all linux distributions.
  2. Uninstall WSL (right-click on the windows start logo> App and Features> Enable or disable windows features> uncheck the box for Windows Subsystem for Linux).
  3. Restart the computer.
  4. Open PowerShell in administrator mode and perform the simplified installation of WSL, as it already (i) enables the optional WSL and Virtual Machine Platform components, (ii) downloads and installs the latest Linux kernel, (iii) defines WSL 2 as the default and (iv) download and install an Ubuntu distribution (Just to inform you, I had previously performed the manual installation):

PS C:\Users\sanam> wsl.exe --install
PS C:\Users\sanam> exit

  1. Restart your computer, and wait while Ubuntu boots.
  2. Within Ubuntu, enter the new user account and password.
  3. Run the commands for installing and configuring the CUDA Toolkit:

$ sudo su

Enter your password to have root access and update Ubuntu.

/home/san# apt update
/home/san# apt upgrade

  1. Then, follow the CUDA Toolkit Documentation > CUDA on WSL tutorial at https://docs.nvidia.com/cuda/wsl-user-guide/index.html, which is the same as the video "GPU Accelerated Machine Learning with WSL 2 "from Youtube:

/home/san# apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub

/home/san# sh -c ‘echo “deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /”> /etc/apt/sources.list.d/cuda.list’

/home/san# apt-get update

/home/san# apt-get install -y cuda-toolkit-11-0

  1. Running a CUDA application:

/home/san# cd /usr/local /cuda/samples/4_Finance/BlackScholes

/usr/local /cuda/samples/4_Finance/BlackScholes# make

/usr/local /cuda/samples/4_Finance/BlackScholes# ./BlackScholes

Ok. All right !!!
Thank you all!
Obrigado a todos!

1 Like

I installed cuda-toolkit-11-2 from the ubuntu2004 repo, and am getting the error you previously had. Maybe the older version is the one required (it’s also what the guide instructs us to install, as you followed)

Uninstalled cuda-toolkit-11-2 and replaced with the older version, cuda-toolkit-11-0, which fixed the problem. Also I don’t think there’s a difference between the 1804 and 2004 repos, at least in regards to the toolkit and its dependencies

Upgrading to cuda-toolkit-11-2 breaks it for me. Downgrading to cuda-toolkit-11-0 fixes it. There’s some sort of change that breaks WSL support in the newer version.

Just pulling info from the other thread on the devicequery sample, I was mucking around with it and there’s something that’s not working in the latest 11-2, but i’m running 11-1 without issue (posted this over on https://forums.developer.nvidia.com/t/devicequery-sample-fails-to-run/163871/8?u=anthony.tan)

After 2-3 days of trouble shooting, this finally worked for me! Specifically using apt-get install -y cuda-toolkit-11-0 instead of $ apt-get install -y cuda. This was confusing, because the instructions here which specifically target WSL, says only cuda, but this is wrong.

I first had to uninstall cuda-11.2, using the instructions found here. I am on WSL, but used the instructions for Ubuntu (then finally manually deleted the /usr/local/cuda-11.2 directory).

I had also forgot to add the following lines to my .bashrc:
export PATH="/usr/local/cuda-11.0/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-11.0/lib64:$LD_LIBRARY_PATH"

Ah, yeah, there’s that. If you install the CUDA toolkit 11-2, it’ll install the samples to /usr/local/cuda-11.2 and symlink over /usr/local/cuda to that directory (via /etc/alternatives).

Depending on what you’re doing, instead of PATH patching, you might find it easier to grab the a Makefiles from the samples area for 11-1 and using that - you should see at the top the toolkit is defined with a conditional, so you could do:

make CUDA_PATH=/usr/local/specific/location/of/toolkit

(I haven’t done much playing with this other than getting the samples to work/run so can’t comment on the LD_LIBRARY_PATH thing though)

I did somewhat a similar process. Reinstalled the Linux distro and reinstall. I found in my case I had to reinstall the Nvidia driver again. Until I did that - kept getting the error. No clue why.