Ubuntu cuda-11-8 package wrong dependency on cuda-drivers

Accelerated Computing CUDA CUDA Setup and Installation
,
1

I am trying to install the CUDA 11.8 with driver version 520 on Ubuntu 20.04. It seems that the dependencies of the packages are wrong.
I have setup the package repo in /etc/apt/sources.list.

deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /

sudo apt install cuda-drivers-520 cuda-11-8

Results in

cuda-drivers-520 is already the newest version (520.61.05-1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 cuda-11-8 : Depends: cuda-runtime-11-8 (>= 11.8.0) but it is not going to be installed
             Depends: cuda-demo-suite-11-8 (>= 11.8.86) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

When I try to install CUDA, it is trying to remove the 520 driver and install 525

sudo apt install cuda-11-8

The following packages will be REMOVED:
  cuda-drivers-520 libnvidia-cfg1-520 libnvidia-compute-520 libnvidia-compute-520:i386 libnvidia-decode-520 libnvidia-decode-520:i386 libnvidia-encode-520 libnvidia-encode-520:i386 libnvidia-extra-520 libnvidia-fbc1-520 libnvidia-fbc1-520:i386 libnvidia-gl-520 libnvidia-gl-520:i386 nvidia-compute-utils-520 nvidia-dkms-520
  nvidia-driver-520 nvidia-kernel-common-520 nvidia-kernel-source-520 nvidia-utils-520 xserver-xorg-video-nvidia-520
The following NEW packages will be installed:
  cuda-11-8 cuda-cccl-11-8 cuda-command-line-tools-11-8 cuda-compiler-11-8 cuda-cudart-11-8 cuda-cudart-dev-11-8 cuda-cuobjdump-11-8 cuda-cupti-11-8 cuda-cupti-dev-11-8 cuda-cuxxfilt-11-8 cuda-demo-suite-11-8 cuda-documentation-11-8 cuda-driver-dev-11-8 cuda-drivers cuda-drivers-525 cuda-gdb-11-8 cuda-libraries-11-8
  cuda-libraries-dev-11-8 cuda-memcheck-11-8 cuda-nsight-11-8 cuda-nsight-compute-11-8 cuda-nsight-systems-11-8 cuda-nvcc-11-8 cuda-nvdisasm-11-8 cuda-nvml-dev-11-8 cuda-nvprof-11-8 cuda-nvprune-11-8 cuda-nvrtc-11-8 cuda-nvrtc-dev-11-8 cuda-nvtx-11-8 cuda-nvvp-11-8 cuda-profiler-api-11-8 cuda-runtime-11-8 cuda-sanitizer-11-8
  cuda-toolkit-11-8 cuda-toolkit-11-8-config-common cuda-toolkit-11-config-common cuda-toolkit-config-common cuda-tools-11-8 cuda-visual-tools-11-8 libcublas-11-8 libcublas-dev-11-8 libcufft-11-8 libcufft-dev-11-8 libcufile-11-8 libcufile-dev-11-8 libcurand-11-8 libcurand-dev-11-8 libcusolver-11-8 libcusolver-dev-11-8
  libcusparse-11-8 libcusparse-dev-11-8 libnpp-11-8 libnpp-dev-11-8 libnvidia-cfg1-525 libnvidia-compute-525 libnvidia-compute-525:i386 libnvidia-decode-525 libnvidia-decode-525:i386 libnvidia-encode-525 libnvidia-encode-525:i386 libnvidia-extra-525 libnvidia-fbc1-525 libnvidia-fbc1-525:i386 libnvidia-gl-525
  libnvidia-gl-525:i386 libnvjpeg-11-8 libnvjpeg-dev-11-8 nvidia-compute-utils-525 nvidia-dkms-525 nvidia-driver-525 nvidia-kernel-common-525 nvidia-kernel-source-525 nvidia-utils-525 xserver-xorg-video-nvidia-525

According to CUDA Compatibility :: NVIDIA Data Center GPU Driver Documentation
driver version 520.61.05 should be compatible with CUDA 11.8. Also according to this documentation driver version 525 is not compatible with CUDA 11.8.

2

The solution is

sudo apt install cuda-11-8 cuda-drivers=520.61.05-1

But cuda-runtime-11-8 has the wrong dependency

apt show cuda-runtime-11-8

Package: cuda-runtime-11-8
Version: 11.8.0-1
Priority: optional
Section: multiverse/devel
Maintainer: cudatools <cudatools@nvidia.com>
Installed-Size: 7,168 B
Depends: cuda-libraries-11-8 (>= 11.8.0), cuda-drivers (>= 520.61.05)
Download-Size: 2,424 B
APT-Sources: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
Description: CUDA Runtime 11.8 meta-package
 Meta-package containing all runtime library packages and the CUDA driver.
 Locked at CUDA Toolkit version 11.8.

I think that is also the case for all other CUDA 11 packages.

3

Hi @sogartar

You can always use a newer NVIDIA driver (notice the plus sign on each column). The cuda-compat-X-Y packages are meant for using older drivers with a newer toolkit, something that would otherwise be incompatible.

Here is a table of the meta-package entry points: CUDA Installation Guide for Linux
I believe you want to use

sudo apt-get install cuda-toolkit-11-8 cuda-drivers-520 --verbose-versions

Here is “simplified” diagram of the package dependency graph

cuda
└── cuda-12-0
    ├── cuda-runtime-12-0
    │   └── cuda-drivers
    │       └── cuda-drivers-525
    │           └── nvidia-driver-525
    └── cuda-toolkit-12-0
        ├── cuda-compiler-12-0
        ├── cuda-libraries-dev-12-0
        │   └── cuda-libraries-12-0
        └── cuda-tools-12-0

cuda-11-8
├── cuda-runtime-11-8
│   └── cuda-drivers
│       └── cuda-drivers-525
│           └── nvidia-driver-525
└── cuda-toolkit-11-8
    ├── cuda-compiler-11-8
    ├── cuda-libraries-dev-11-8
    │   └── cuda-libraries-11-8
    └── cuda-tools-11-8

cuda-drivers-520
└── nvidia-driver-520
4

Hi, How would one install the nvidia-driver-toolkit for newer CUDA / Ubuntu versions?

GPU: 4090
Ubuntu: 22.04
Nvidia-driver: 530
CUDA: 12.1

I get the error:
“function cbapi->getCuptiStatus() failed with error CUPTI_ERROR_NOT_INITIALIZED (15)mage(s)]
WARNING:2023-04-19 11:02:44 4368:4368 init.cpp:147] CUPTI initialization failed - CUDA profiler activities will be missing
INFO:2023-04-19 11:02:44 4368:4368 init.cpp:149] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to NVIDIA Development Tools Solutions - CUPTI_ERROR_INSUFFICIENT_PRIVILEGES: CUPTI Permission issue with Performance Counters | NVIDIA Developer”

while trying to run torch.profile. I have added the " options nvidia "NVreg_RestrictProfilingToAdminUsers=0"" option to a .conf file and confirmed that any user can now access the profiler without sudo permissions. However, I still get the CUPTI error which must mean that while CUDA is installed the nvidia-cuda-toolkit can’t be?

I also don’t get anything when trying to ‘nvcc --version’ check in my terminal other than ‘plese install the nvidia-cuda-toolkit’

How do I resolve this?

5

After executing this command and rebooting the computer, in ubuntu 20.04 and running the command nvidia-smi

It showed me it installed driver 525.105.17 and cuda version was set to 12.0

This is not good enough, I require cuda 11.8 and an identical computer running GeForce RTX 3060 was able to install the drivers, however replicating it usually results in a black screen where the login screen never appears. I was hoping this would be different.

6

Hi @woodconn
On a clean system, I cannot reproduce CUDA 12.0/525 installation, it is installs CUDA 11.8/520 (with exception of nvidia-modprobe and nvidia-settings)

Following the pre-installation actions

uname -m && cat /etc/*release
gcc --version
sudo apt-get install linux-headers-$(uname -r) --verbose-versions

and package manager installation section

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install cuda-toolkit-11-8 cuda-drivers-520 --verbose-versions

Regarding the black screen, make sure to disable UEFI SecureBoot.

7

Hi,
unfortunately I faced exactly the same issue. I really needed cuda 11.8, since I wasn’t able to get backwards on WSL I even created a linux partition and installed “real” linux just in order to be able to use 11.8 and after following the instruction:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pinsudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.debsudo dpkg -i cuda-repo-ubuntu2204-11-8-local_11.8.0-520.61.05-1_amd64.debsudo cp /var/cuda-repo-ubuntu2204-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/sudo apt-get updatesudo apt-get -y install cuda

I ended up with cudatoolkit 12.2. Any idea why? It automatically swiched to 12.2. and nvidia drivers 535 instead of 520 and I want to cry because I tried to be really careful to avoid exactly this situation.

Before that i tried the way you posted here but it always ended with an error:

sudo apt-get install cuda-toolkit-11-8 cuda-drivers-520 --verbose-versions
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
cuda-drivers-520 is already the newest version (520.61.05-1).
cuda-drivers-520 set to manually installed.
cuda-toolkit-11-8 is already the newest version (11.8.0-1).
cuda-toolkit-11-8 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
8 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Setting up nvidia-dkms-520 (520.61.05-0ubuntu1) …
update-initramfs: deferring update (trigger activated)

A modprobe blacklist file has been created at /etc/modprobe.d to prevent Nouveau
from loading. This can be reverted by deleting the following file:
/etc/modprobe.d/nvidia-graphics-drivers.conf

A new initrd image has also been created. To revert, please regenerate your
initrd by running the following command after deleting the modprobe.d file:
/usr/sbin/initramfs -u

*** Reboot your computer and verify that the NVIDIA graphics driver can ***
*** be loaded. ***

INFO:Enable nvidia
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/dell_latitude
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/put_your_quirks_here
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/lenovo_thinkpad
Removing old nvidia-520.61.05 DKMS files…
Deleting module nvidia-520.61.05 completely from the DKMS tree.
Loading new nvidia-520.61.05 DKMS files…
Building for 6.2.0-26-generic
Building for architecture x86_64
Building initial module for 6.2.0-26-generic
ERROR: Cannot create report: [Errno 17] File exists: ‘/var/crash/nvidia-dkms-520.0.crash’
Error! Bad return status for module build on kernel: 6.2.0-26-generic (x86_64)
Consult /var/lib/dkms/nvidia/520.61.05/build/make.log for more information.
dpkg: error processing package nvidia-dkms-520 (–configure):
installed nvidia-dkms-520 package post-installation script subprocess returned error exit status 10
dpkg: dependency problems prevent configuration of cuda-drivers-520:
cuda-drivers-520 depends on nvidia-dkms-520 (>= 520.61.05); however:
Package nvidia-dkms-520 is not configured yet.

dpkg: error processing package cuda-drivers-520 (–configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of cuda-drivers:
cuda-drivers depends on cuda-drivers-520 (= 520.61.05-1); however:
Package cuda-drivers-520 is not configured yet.

dpkg: error processing package cuda-drivers (–configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of nvidia-driver-520:
nvidia-driver-520 depends on nvidia-dkms-520 (= 520.61.05-0ubuntu1); however:
Package nvidia-dkms-520 is not configured yet.

dpkg: error processing package nvidia-driver-520 (–configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of cuda-runtime-11-8:
cuda-runtime-11-8 depends on cuda-drivers (>No apport report written because the error message indicates its a followup error from a previous failure.
No apport report written because the error message indicates its a followup error from a previous failure.
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
No apport report written because MaxReports is reached already
= 520.61.05); however:
Package cuda-drivers is not configured yet.

dpkg: error processing package cuda-runtime-11-8 (–configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of cuda-11-8:
cuda-11-8 depends on cuda-runtime-11-8 (>= 11.8.0); however:
Package cuda-runtime-11-8 is not configured yet.

dpkg: error processing package cuda-11-8 (–configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of cuda-demo-suite-11-8:
cuda-demo-suite-11-8 depends on cuda-runtime-11-8; however:
Package cuda-runtime-11-8 is not configured yet.

dpkg: error processing package cuda-demo-suite-11-8 (–configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of cuda:
cuda depends on cuda-11-8 (>= 11.8.0); however:
Package cuda-11-8 is not configured yet.

dpkg: error processing package cuda (–configure):
dependency problems - leaving unconfigured
Processing triggers for initramfs-tools (0.140ubuntu13.4) …
update-initramfs: Generating /boot/initrd.img-6.2.0-26-generic
Errors were encountered while processing:
nvidia-dkms-520
cuda-drivers-520
cuda-drivers
nvidia-driver-520
cuda-runtime-11-8
cuda-11-8
cuda-demo-suite-11-8
cuda
E: Sub-process /usr/bin/dpkg returned an error code (1)

8

Hi @stepien.natalia
Based on the output provided, it’s possible that the 520.61.05 driver is not compatible with the Linux kernel 6.2.0-26-generic

A couple of options I suggest trying

  • Try using an older 5.x Linux kernel if available (looks like it might not be for Ubuntu 22.04)
  • Upgrade to the newer NVIDIA driver (i.e. 535) but continue using CUDA Toolkit 11.8 (“backwards compatibility”)