I’m trying to run a GPU-enabled llama app, but getting the following error:
CUDA error 100 at /tmp/pip-install-n1njvklt/llama-cpp-python_9dd25f9be6684ee0ba46db57165163dd/vendor/llama.cpp/ggml-cuda.cu:5572: no CUDA-capable device is detected
current device: 32703
I followed the instructions listed in 1. NVIDIA GPU Accelerated Computing on WSL 2 — CUDA on WSL 12.3 documentation, which seem to amount to:
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.3.0/local_installers/cuda-repo-wsl-ubuntu-12-3-local_12.3.0-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-12-3-local_12.3.0-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-12-3-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get -y install cuda-toolkit-12-3
and below is what my WSL system is currently showing. What am I missing/doing wrong here?
root@mymachine:/mnt/c/dev# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0
root@mymachine:/mnt/c/dev/git/github_/privateGPT# nvidia-smi
Wed Nov 1 13:28:50 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01 Driver Version: 545.84 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro M1000M On | 00000000:01:00.0 Off | N/A |
| N/A 0C P0 N/A / 200W | 746MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 23 G /Xwayland N/A |
+---------------------------------------------------------------------------------------+
Edition Windows 10 Pro
Version 22H2
Installed on 10/18/2023
OS build 19045.3636
Experience Windows Feature Experience Pack 1000.19053.1000.0