Ubuntu cuda-11-8 package wrong dependency on cuda-drivers

I am trying to install the CUDA 11.8 with driver version 520 on Ubuntu 20.04. It seems that the dependencies of the packages are wrong.
I have setup the package repo in /etc/apt/sources.list.

deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /
sudo apt install cuda-drivers-520 cuda-11-8

Results in

cuda-drivers-520 is already the newest version (520.61.05-1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 cuda-11-8 : Depends: cuda-runtime-11-8 (>= 11.8.0) but it is not going to be installed
             Depends: cuda-demo-suite-11-8 (>= 11.8.86) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

When I try to install CUDA, it is trying to remove the 520 driver and install 525

sudo apt install cuda-11-8
The following packages will be REMOVED:
  cuda-drivers-520 libnvidia-cfg1-520 libnvidia-compute-520 libnvidia-compute-520:i386 libnvidia-decode-520 libnvidia-decode-520:i386 libnvidia-encode-520 libnvidia-encode-520:i386 libnvidia-extra-520 libnvidia-fbc1-520 libnvidia-fbc1-520:i386 libnvidia-gl-520 libnvidia-gl-520:i386 nvidia-compute-utils-520 nvidia-dkms-520
  nvidia-driver-520 nvidia-kernel-common-520 nvidia-kernel-source-520 nvidia-utils-520 xserver-xorg-video-nvidia-520
The following NEW packages will be installed:
  cuda-11-8 cuda-cccl-11-8 cuda-command-line-tools-11-8 cuda-compiler-11-8 cuda-cudart-11-8 cuda-cudart-dev-11-8 cuda-cuobjdump-11-8 cuda-cupti-11-8 cuda-cupti-dev-11-8 cuda-cuxxfilt-11-8 cuda-demo-suite-11-8 cuda-documentation-11-8 cuda-driver-dev-11-8 cuda-drivers cuda-drivers-525 cuda-gdb-11-8 cuda-libraries-11-8
  cuda-libraries-dev-11-8 cuda-memcheck-11-8 cuda-nsight-11-8 cuda-nsight-compute-11-8 cuda-nsight-systems-11-8 cuda-nvcc-11-8 cuda-nvdisasm-11-8 cuda-nvml-dev-11-8 cuda-nvprof-11-8 cuda-nvprune-11-8 cuda-nvrtc-11-8 cuda-nvrtc-dev-11-8 cuda-nvtx-11-8 cuda-nvvp-11-8 cuda-profiler-api-11-8 cuda-runtime-11-8 cuda-sanitizer-11-8
  cuda-toolkit-11-8 cuda-toolkit-11-8-config-common cuda-toolkit-11-config-common cuda-toolkit-config-common cuda-tools-11-8 cuda-visual-tools-11-8 libcublas-11-8 libcublas-dev-11-8 libcufft-11-8 libcufft-dev-11-8 libcufile-11-8 libcufile-dev-11-8 libcurand-11-8 libcurand-dev-11-8 libcusolver-11-8 libcusolver-dev-11-8
  libcusparse-11-8 libcusparse-dev-11-8 libnpp-11-8 libnpp-dev-11-8 libnvidia-cfg1-525 libnvidia-compute-525 libnvidia-compute-525:i386 libnvidia-decode-525 libnvidia-decode-525:i386 libnvidia-encode-525 libnvidia-encode-525:i386 libnvidia-extra-525 libnvidia-fbc1-525 libnvidia-fbc1-525:i386 libnvidia-gl-525
  libnvidia-gl-525:i386 libnvjpeg-11-8 libnvjpeg-dev-11-8 nvidia-compute-utils-525 nvidia-dkms-525 nvidia-driver-525 nvidia-kernel-common-525 nvidia-kernel-source-525 nvidia-utils-525 xserver-xorg-video-nvidia-525

According to CUDA Compatibility :: NVIDIA Data Center GPU Driver Documentation
driver version 520.61.05 should be compatible with CUDA 11.8. Also according to this documentation driver version 525 is not compatible with CUDA 11.8.

The solution is

sudo apt install cuda-11-8 cuda-drivers=520.61.05-1

But cuda-runtime-11-8 has the wrong dependency

apt show cuda-runtime-11-8
Package: cuda-runtime-11-8
Version: 11.8.0-1
Priority: optional
Section: multiverse/devel
Maintainer: cudatools <cudatools@nvidia.com>
Installed-Size: 7,168 B
Depends: cuda-libraries-11-8 (>= 11.8.0), cuda-drivers (>= 520.61.05)
Download-Size: 2,424 B
APT-Sources: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages
Description: CUDA Runtime 11.8 meta-package
 Meta-package containing all runtime library packages and the CUDA driver.
 Locked at CUDA Toolkit version 11.8.

I think that is also the case for all other CUDA 11 packages.

Hi @sogartar

You can always use a newer NVIDIA driver (notice the plus sign on each column). The cuda-compat-X-Y packages are meant for using older drivers with a newer toolkit, something that would otherwise be incompatible.

Here is a table of the meta-package entry points: CUDA Installation Guide for Linux
I believe you want to use

sudo apt-get install cuda-toolkit-11-8 cuda-drivers-520 --verbose-versions

Here is “simplified” diagram of the package dependency graph

cuda
└── cuda-12-0
    ├── cuda-runtime-12-0
    │   └── cuda-drivers
    │       └── cuda-drivers-525
    │           └── nvidia-driver-525
    └── cuda-toolkit-12-0
        ├── cuda-compiler-12-0
        ├── cuda-libraries-dev-12-0
        │   └── cuda-libraries-12-0
        └── cuda-tools-12-0

cuda-11-8
├── cuda-runtime-11-8
│   └── cuda-drivers
│       └── cuda-drivers-525
│           └── nvidia-driver-525
└── cuda-toolkit-11-8
    ├── cuda-compiler-11-8
    ├── cuda-libraries-dev-11-8
    │   └── cuda-libraries-11-8
    └── cuda-tools-11-8

cuda-drivers-520
└── nvidia-driver-520