Nvidia Cuda Compiler not showing up in Linux 22.04

matteo.ungaro · April 24, 2022, 2:17pm

Hi there,

I recently installed Ubuntu 22.04 “Jammy Jellyfish” and this is what my shell says when I run nvidia-smi:

Seems I have the latest version for the drivers (510.60.02) as well as the latest CUDA Version (11.6).

However, here comes the problem. When I run nvcc -V, I’m prompted with the message:

Command ‘nvcc’ not found, but can be installed with:
sudo apt install nvidia-cuda-toolkit

Now, from my understanding if the output of nvidia-smi shows a CUDA version it menas that the CUDA Toolkit has been installed, correct me if I’m wrong.

Strangely, I cannot find the cuda-11.6 folder anywhere in my root directory, so even after adding/exporting the path in the ~/ .bashcr the command nvcc -V doesn’t show anything.

The issue is that I really need the CUDA Toolkit to work properly since I have then to install PyCUDA in order to run a scaffolding tool for genetic analyses which makes use of GPU acceleration.

Another thing is the Nvidia X Server Settings, I’m currently able to fully access all features from the GUI:

On the webpage of the tool I’m planning to use, they redirect users to this page for installing the latest version of the NVIDIA CUDA toolkit: https://developer.nvidia.com/cuda-downloads, albeit it shows only compatibility with versions up to Ubuntu 20.04.

I would like to know whether it is safe to install this version of the NVIDIA CUDA toolkit even though I’m running Ubuntu 22.04 without having my Nvidia X Server Settings compromised. In fact, for some weird reasons with the previous version of Ubuntu I had, after installing the NVIDIA CUDA toolkit following this procedure, I had some troubles with my NVIDIA Settings (see image) which prevented me from accessing all features, such as System Information, X Server Display Configuration and XScreen 0.

Please, let me know what should I do and whether it is safe to install the NVIDIA CUDA toolkit form such a link without messing up my current Nvidia X Server Settings. Thanks in advance.

matteo.ungaro · April 24, 2022, 2:21pm

CONTINUATION – SEE BEFORE – couldn’t add more than one media as new user
Another thing is the Nvidia X Server Settings, I’m currently able to fully access all features from the GUI:

On the webpage of the tool I’m planning to use, they redirect users to this page for installing the latest version of the NVIDIA CUDA toolkit: https://developer.nvidia.com/cuda-downloads, albeit it shows only compatibility with versions up to Ubuntu 20.04.

I would like to know whether it is safe to install this version of the NVIDIA CUDA toolkit even though I’m running Ubuntu 22.04 without having my Nvidia X Server Settings compromised. In fact, for some weird reasons with the previous version of Ubuntu I had, after installing the NVIDIA CUDA toolkit following this procedure, I had some troubles with my NVIDIA Settings (see image) which prevented me from accessing all features, such as System Information, X Server Display Configuration and XScreen 0.

Please, let me know what should I do and whether it is safe to install the NVIDIA CUDA toolkit form such a link without messing up my current Nvidia X Server Settings. Thanks in advance.

generix · April 25, 2022, 7:46am

Follow the steps on the download page but omit thelast step, don’t install “cuda”. Instead, only install the toolkit
sudo apt install cuda-toolkit

nvidia-smi only displays cuda driver compatibility level, not the installed toolkit version.

matteo.ungaro · April 25, 2022, 10:36am

Hi,

Thanks a lot. However, now when running sudo apt-get update I got this message:

W: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

I know it is something which can be fixed (in fact I did it with Dropbox) but I don’y know how to do it in this case. Please, let me know the procedure to get rid of this message.

Thanks in advance.

P.S. I installed the toolkit with the following command sudo apt install nvidia-cuda-toolkit since the sudo apt install cuda-toolkit returned this error:

E: Unable to locate package cuda-toolkit

matteo.ungaro · April 25, 2022, 10:42am

Also, I couldn’t run anymore the nvidia-smi command and I was prompted with an indication to re-install it. After I did it now I have this error message showing…

Failed to initialize NVML: Driver/library version mismatch

How can it be fixed?

Another issue is related to the Software and Updates app, it show only this option without the possibility to revert to any other alternative:

Please, let me know what to do I really want to avoid to re-install the OS or anything related to the Nvidia Settings that can cause me issues as before.

Thanks in advance.

generix · April 25, 2022, 11:04am

apt-key:
https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html
nvidia-cuda-toolkit is a repackaging by debian/ubuntu, I don’t know anything about it.

The NVML message points to having two driver versions installed over oneanother.
Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.

matteo.ungaro · April 25, 2022, 11:16am

Thanks again,

I though that was the issue, so I proceeded as indicated in this guide: CUDA Installation Guide for Linux.

Basically, I removed all related NVIDIA software and drivers and started again with a clean installation form the Software and Updates app. The good news is that it worked, the situation is back to normal!

However, despite my nvidia-smi showing the same exact configuration as before, I’m now facing the same exact issue when running nvcc -V. I got prompted the following message:

Command ‘nvcc’ not found, but can be installed with:
sudo apt install nvidia-cuda-toolkit

matteo.ungaro · April 25, 2022, 11:19am

FOLLOW UP – SEE BEFORE-- as a new user I cannot post more than one link per message
Now, my question is it safe to install the CUDA tollkit following the former command instead of relying on the procedure here described CUDA Toolkit 11.7 Update 1 Downloads | NVIDIA Developer?

Thanks in advance.

johnnynuca14 · April 25, 2022, 1:34pm

Yeah I have the same problem. Nvidia must launch a new driver version to support kernel 5.18 because that not working and new support for wayland by default and adding ubuntu 22.04 repo. Also, nvidia must modify documentation to support new apt keys. I solve it installing:
sudo add-apt-repository ppa:graphics-drivers/ppa

sudo apt-get update

sudo ubuntu-drivers autoinstall

reboot

and install cuda 11.6 from:
https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=runfile_local

and removing that it install the nvidia drivers.

matteo.ungaro · April 26, 2022, 10:29am

Hi John,

Thanks a lot for the info. By the way, are you using Ubuntu 22.04? In fact, I was afraid to mess around too much in trying to do things which might have broken the system.

So, I’m assuming the procedure detailed it is something you have tested on your own version of “Jammy Jellyfish”, am I correct?

Thanks in advance.

P.S. if you know it what would be the difference form installing the CUDA Toolkit from the NVIDIA repository (as indicated in you message) instead of using the repackaging from Debian/Ubuntu (sudo apt install nvidia-cuda-toolkit)

coryg89 · April 27, 2022, 9:15am

I ran into this problem after installing 22.04 earlier as well. I can confirm that johnnynuca14’s solution posted above does indeed work on 22.04. I already had the 510 driver installed, same as you, so all I had to do was run the runfile (local) installer by executing the following:

user1@ubuntu2204:~$ wget https://developer.download.nvidia.com/compute/cuda/11.6.2/local_installers/cuda_11.6.2_510.47.03_linux.run
user1@ubuntu2204:~$ sudo sh cuda_11.6.2_510.47.03_linux.run

per the instructions provided by nvidia here, and then selecting the option to tell it not to install the included driver as shown in johnnynuca14’s screenshot above. After doing this and adding /usr/local/cuda/bin to my PATH I was able to execute nvcc successfully as shown below:

user1@ubuntu2204:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

matteo.ungaro · April 27, 2022, 9:20am

Thanks to both,

I actually just wrote to Johnny in private and I apologize for that. I will closely follow the commands you indicated and let you know.

Thanks again!

johnnynuca14 · April 27, 2022, 9:29am

yeah i’m using ubuntu 22.04 release

matteo.ungaro · April 27, 2022, 10:39am

@johnnynuca14 & @coryg89, everything works perfectly!!! Thanks a lot guys.

ksimady · April 28, 2022, 3:50pm

According to what is posted here I am in the same situation, with the 510.60.02 driver installed on Ubuntu 22.04.

I have followed the method presented here by first launching :

sudo add-apt-repository ppa:graphics-drivers/ppa

and then downloading and running the .run file. But the install did not complete, stating that the driver is too old:

= Summary =
===========

Driver:   Not Selected
Toolkit:  Installed in /usr/local/cuda-11.6/

Please make sure that
 -   PATH includes /usr/local/cuda-11.6/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-11.6/lib64, or, add /usr/local/cuda-11.6/lib64 to /etc/ld.so.conf and run ldconfig as root

To uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-11.6/bin
***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 510.00 is required for CUDA 11.6 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
    sudo <CudaInstaller>.run --silent --driver

Logfile is /var/log/cuda-installer.log```


When I run nvidia-smi:

```Thu Apr 28 17:31:43 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.60.02    Driver Version: 510.60.02    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   47C    P8     9W / 170W |      8MiB / 12288MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2017      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A      2251      G   ...ome-remote-desktop-daemon        2MiB |
+-----------------------------------------------------------------------------+```

Has someone encountered similar difficulties?

johnnynuca14 · May 2, 2022, 9:52am

sudo apt-get remove --purge '^nvidia-.*'
sudo apt purge '*nvidia*'
sudo apt autoremove

then install with

sudo ubuntu-drivers autoinstall

and reboot.
Then install cuda without drivers and put in .bashrc

export PATH=/usr/local/cuda-11.6/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.6/lib64/${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

I’m in ubuntu 22.04 with kernel 5.17.5. With kernel 5.18 use new C version and it is not comptatible…
My hardware is: 3090 FE, 12900k and DDR5 6400mhz 32gb cl32 g.skill z5

ksimady · May 2, 2022, 10:17am

Thank you for your answer. I have followed your steps up to the driver auto-install which all performed successfully.

My hardware is a bit older (3060 Lite Hash, 10900K, DDR4 3200mhz 64gb), and so is my kernel version (Linux 5.15.0-27-generic).

Should I try to install an upgraded kernel version before trying to install the cuda drivers, or should the one I have be enough?

johnnynuca14 · May 2, 2022, 10:24am

Kernel 5.18 rc5

BUG: 88hz only…

johnnynuca14 · May 2, 2022, 10:26am

To upgrade kernel. You have to follow this steps:

Upgrade with

https://github.com/bkw777/mainline
sudo add-apt-repository ppa:cappelikan/ppa
sudo apt update
sudo apt install mainline

install 5.17.5
then reboot. Boot with new kernel and:

sudo apt-get remove --purge '^nvidia-.*'
sudo apt purge '*nvidia*'
sudo apt autoremove

Then remove old kernel button in mainline app. (Delete 5.15)
After this,

sudo ubuntu-drivers autoinstall

reboot and then install cuda as you seen above

This is because kms nvidia not install automatically on 5.17.5, I don’t know why…

ksimady · May 3, 2022, 9:57am

Thank’s for all the help.

After upgrading the kernel to version 5.17.5 I tried the same steps again using the runfile.

Interestingly the same warning stating that the installation was incomplete and the driver version not high enough appeared.

But I have still done some tests (installed cupy in a virtual environment) and it seems that the matrix operations in fact run on the GPU.

So even with this message cuda can still be installed and used.