Possible conflicting packages when upgrading CUDA on Ubuntu 18.04 LTS

Whilst attempting to upgrade my CUDA install on Ubuntu 18.04 LTS I have encountered a problem - it seems I have an issue with conflicting packages, which is preventing another package being installed, although I am not sure exactly which packages these are, or whether it’s safe to remove one of them. I began the process by removing the previous CUDA install, following the instructions provided here:

sudo rm /etc/apt/sources.list.d/cuda*
sudo apt remove --autoremove nvidia-cuda-toolkit
sudo apt remove --autoremove nvidia-*

sudo apt-get purge nvidia*
sudo apt-get autoremove
sudo apt-get autoclean

sudo rm -rf /usr/local/cuda*

Then I attempted to install the latest CUDA, using the TensorFlow instructions here (which I had followed for the previous CUDA install):

sudo apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb

But after executing that I got the following error:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies.
 libnvidia-extra-450 : Depends: libnvidia-extra-460 but it is not installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

Then trying sudo apt --fix-broken install gave me:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  g++-6 libaccinj64-9.1 libcublas9.1 libcudart9.1 libcufft9.1 libcufftw9.1 libcuinj64-9.1 libcurand9.1 libcusolver9.1
  libcusparse9.1 libnppc9.1 libnppial9.1 libnppicc9.1 libnppicom9.1 libnppidei9.1 libnppif9.1 libnppig9.1 libnppim9.1
  libnppist9.1 libnppisu9.1 libnppitc9.1 libnpps9.1 libnvblas9.1 libnvgraph9.1 libnvidia-common-440 libnvidia-extra-440
  libnvrtc9.1 libnvtoolsext1 libnvvm3 libstdc++-6-dev libthrust-dev libvdpau-dev nvidia-cuda-doc nvidia-cuda-gdb
  nvidia-opencl-dev nvidia-profiler nvidia-visual-profiler ocl-icd-opencl-dev opencl-c-headers
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libnvidia-extra-460
The following NEW packages will be installed
  libnvidia-extra-460
0 to upgrade, 1 to newly install, 0 to remove and 273 not to upgrade.
2 not fully installed or removed.
Need to get 0 B/40.9 kB of archives.
After this operation, 120 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
(Reading database ... 196344 files and directories currently installed.)
Preparing to unpack .../libnvidia-extra-460_460.106.00-0ubuntu1_amd64.deb ...
Unpacking libnvidia-extra-460:amd64 (460.106.00-0ubuntu1) ...
dpkg: error processing archive /var/cache/apt/archives/libnvidia-extra-460_460.106.00-0ubuntu1_amd64.deb (--unpack):
 trying to overwrite '/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so', which is also in package libnvidia-compute-450:amd64 450.36.06-0ubuntu1
Errors were encountered while processing:
 /var/cache/apt/archives/libnvidia-extra-460_460.106.00-0ubuntu1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

I encountered a problem similar to this a few weeks ago, which was solved by removing the package containing the file that was being overwritten. In this case I suspect there is a driver conflict, since the packages in question contain driver numbers (libnvidia-compute-450, libnvidia-extra-460). But I’m not sure. What exactly is libnvidia-extra-X, anyway? Following the suggested fix from this issue I am minded to run:

sudo dpkg -P nvidia-driver-450

Or force the install with:

sudo dpkg -i --force-overwrite /var/cache/apt/archives/libnvidia-compute-450:amd64 450.36.06-0ubuntu1
sudo apt -f install

The output of dpkg -l | grep -i nvidia is:

ii  cuda-nsight-compute-10-1                   10.1.243-1                                       amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-10-1                   10.1.243-1                                       amd64        NVIDIA Nsight Systems
ii  cuda-nvtx-10-1                             10.1.243-1                                       amd64        NVIDIA Tools Extension
ii  libaccinj64-9.1:amd64                      9.1.85-3ubuntu1                                  amd64        NVIDIA ACCINJ Library (64-bit)
ii  libcublas9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA cuBLAS Library
ii  libcudart9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA CUDA Runtime Library
ii  libcufft9.1:amd64                          9.1.85-3ubuntu1                                  amd64        NVIDIA cuFFT Library
ii  libcufftw9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA cuFFTW Library
ii  libcuinj64-9.1:amd64                       9.1.85-3ubuntu1                                  amd64        NVIDIA CUINJ Library (64-bit)
ii  libcurand9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA cuRAND Library
ii  libcusolver9.1:amd64                       9.1.85-3ubuntu1                                  amd64        NVIDIA cuSOLVER Library
ii  libcusparse9.1:amd64                       9.1.85-3ubuntu1                                  amd64        NVIDIA cuSPARSE Library
ii  libnppc9.1:amd64                           9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives core runtime library
ii  libnppial9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Arithmetic and Logic
ii  libnppicc9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Color Conversion
ii  libnppicom9.1:amd64                        9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Compression
ii  libnppidei9.1:amd64                        9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Data Exchange and Initialization
ii  libnppif9.1:amd64                          9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Filters
ii  libnppig9.1:amd64                          9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Geometry transforms
ii  libnppim9.1:amd64                          9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Morphological operations
ii  libnppist9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Statistics
ii  libnppisu9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Support
ii  libnppitc9.1:amd64                         9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives lib for Image Threshold and Compare
ii  libnpps9.1:amd64                           9.1.85-3ubuntu1                                  amd64        NVIDIA Performance Primitives for signal processing runtime library
ii  libnvgraph9.1:amd64                        9.1.85-3ubuntu1                                  amd64        NVIDIA Graph Analytics library (nvGRAPH)
ii  libnvidia-cfg1-450:amd64                   450.36.06-0ubuntu1                               amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-440                       450.119.03-0ubuntu0.18.04.1                      all          Transitional package for libnvidia-common-450
ii  libnvidia-common-450                       460.91.03-0ubuntu0.18.04.1                       all          Transitional package for libnvidia-common-460
ii  libnvidia-common-460                       460.91.03-0ubuntu0.18.04.1                       all          Shared files used by the NVIDIA libraries
rc  libnvidia-compute-440:amd64                440.82-0ubuntu0~0.18.04.2                        amd64        NVIDIA libcompute package
ii  libnvidia-compute-450:amd64                450.36.06-0ubuntu1                               amd64        NVIDIA libcompute package
ii  libnvidia-decode-450:amd64                 450.36.06-0ubuntu1                               amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-450:amd64                 450.36.06-0ubuntu1                               amd64        NVENC Video Encoding runtime library
iU  libnvidia-extra-440:amd64                  450.119.03-0ubuntu0.18.04.1                      amd64        Transitional package for libnvidia-extra-450
iU  libnvidia-extra-450:amd64                  460.91.03-0ubuntu0.18.04.1                       amd64        Transitional package for libnvidia-extra-460
ii  libnvidia-fbc1-450:amd64                   450.36.06-0ubuntu1                               amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-450:amd64                     450.36.06-0ubuntu1                               amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-ifr1-450:amd64                   450.36.06-0ubuntu1                               amd64        NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  libnvrtc9.1:amd64                          9.1.85-3ubuntu1                                  amd64        CUDA Runtime Compilation (NVIDIA NVRTC Library)
ii  libnvtoolsext1:amd64                       9.1.85-3ubuntu1                                  amd64        NVIDIA Tools Extension Library
ii  libnvvm3:amd64                             9.1.85-3ubuntu1                                  amd64        NVIDIA NVVM Library
ii  nsight-compute-2020.1.0                    2020.1.0.33-1                                    amd64        NVIDIA Nsight Compute
rc  nvidia-compute-utils-440                   440.82-0ubuntu0~0.18.04.2                        amd64        NVIDIA compute utilities
ii  nvidia-compute-utils-450                   450.36.06-0ubuntu1                               amd64        NVIDIA compute utilities
ii  nvidia-cuda-doc                            9.1.85-3ubuntu1                                  all          NVIDIA CUDA and OpenCL documentation
ii  nvidia-cuda-gdb                            9.1.85-3ubuntu1                                  amd64        NVIDIA CUDA Debugger (GDB)
rc  nvidia-dkms-440                            440.82-0ubuntu0~0.18.04.2                        amd64        NVIDIA DKMS package
ii  nvidia-dkms-450                            450.36.06-0ubuntu1                               amd64        NVIDIA DKMS package
ii  nvidia-driver-450                          450.36.06-0ubuntu1                               amd64        NVIDIA driver metapackage
rc  nvidia-kernel-common-440                   440.82-0ubuntu0~0.18.04.2                        amd64        Shared files used with the kernel module
ii  nvidia-kernel-common-450:amd64             460.91.03-0ubuntu0.18.04.1                       amd64        Transitional package for nvidia-kernel-common-460
ii  nvidia-kernel-common-460                   460.91.03-0ubuntu0.18.04.1                       amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-450                   450.36.06-0ubuntu1                               amd64        NVIDIA kernel source package
ii  nvidia-machine-learning-repo-ubuntu1804    1.0.0-1                                          amd64        nvidia-machine-learning repository configuration files
ii  nvidia-modprobe                            450.36.06-0ubuntu1                               amd64        Load the NVIDIA kernel driver and create device files
ii  nvidia-opencl-dev:amd64                    9.1.85-3ubuntu1                                  amd64        NVIDIA OpenCL development files
ii  nvidia-profiler                            9.1.85-3ubuntu1                                  amd64        NVIDIA Profiler for CUDA and OpenCL
ii  nvidia-settings                            470.57.01-0ubuntu0.18.04.1                       amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-450                           450.36.06-0ubuntu1                               amd64        NVIDIA driver support binaries
ii  nvidia-visual-profiler                     9.1.85-3ubuntu1                                  amd64        NVIDIA Visual Profiler for CUDA and OpenCL
ii  xserver-xorg-video-nvidia-450              450.36.06-0ubuntu1                               amd64        NVIDIA binary Xorg driver

I tried explicitly removing and purging the libnvidia-compute-450 package, but that did nothing. Not sure what to do, tempted to just remove that driver but having read of people being stung by similar problems I’m wary, don’t want to have to completely reinstall Ubuntu!

OK in the end I managed to fix this by forcing overwrite of the offending package (purging it didn’t work). I ran

sudo dpkg -i --force-overwrite /var/cache/apt/archives/libnvidia-extra-460_460.106.00-0ubuntu1_amd64.deb

Then I was able to successfully run the rest of the TensorFlow CUDA install instructions. Annoyingly, these instructions actually install CUDA 11.0 for TF 2.4 (why?), which are not the current versions of either - it’s easy to miss this!