Cannot get nvidia driver (520, 515, 515-open, or 510) working in Ubuntu 22.10

@christianboin > module: x86/modules: Skipping invalid relocation target
Your build system is broken, producing broken kernel modules. Did you install a different linker, e.g. mold instead of gnu ld?

@generix Not that I know of. I had this same error in Ubuntu 22.04, but didnt think anything of it because I was able to get the drivers to work, albeit it was a huge pain.

Is there a fix for something like this?

First measure would be remove and reinstall kernel headers, then use dkms to revove /install the modules and use modprobe to see if they load.

Sorry I am a bit new to this stuff. Reinstalling kernel headers is:

‘sudo apt install —reinstall linux-headers-$(uname -r)’

To remove kernel modules, do you mean make sure no nvidia modules appear with ‘dkms status’

And lastly, to check with modprobe…

‘sudo modprobe nvidia’

I assume I should just reinstall the 520 driver through the ‘additional drivers tab’ and then do all these steps before trying a reboot?

@generix

kernel headers/modprobe, like you said.
dkms to recompile nvidia modules
dkms status
to see what modules are installed, if none, that’s bad
sudo dkms remove nvidia/(version number)
to remove followed by
sudo dkms install nvidia/(version number)
to recompile the modules.

@generix and I should install the driver first obviously, through the additional drivers tab in ubuntu? Then run all your steps, or reboot and then run the steps from the recovery mode (since my desktop GUI will not load)

Install the driver but don’t reboot. Then check if you can modprobe the driver, if not do the other steps.

OK @generix So I did the following.

  1. install nvidia 520.56.06 from the additional drivers tab
  2. sudo modprobe nvidia → returns => modprobe: ERROR: could not insert 'nvidia': Exec format error
  3. sudo apt install --reinstall linux-headers-$(uname -r) → returns →
Reading package lists... Done                                                                                                               
Building dependency tree... Done                                                                                                            
Reading state information... Done                                                                                                           
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.                                                               
Need to get 3,270 kB of archives.                                                                                                           
After this operation, 0 B of additional disk space will be used.                                                                            
Get:1 http://ca.archive.ubuntu.com/ubuntu kinetic-updates/main amd64 linux-headers-5.19.0-23-generic amd64 5.19.0-23.24 [3,270 kB]          
Fetched 3,270 kB in 0s (8,824 kB/s)                                                                                                         
(Reading database ... 523239 files and directories currently installed.)                                                                    
Preparing to unpack .../linux-headers-5.19.0-23-generic_5.19.0-23.24_amd64.deb ...                                                          
Unpacking linux-headers-5.19.0-23-generic (5.19.0-23.24) over (5.19.0-23.24) ...                                                            
Setting up linux-headers-5.19.0-23-generic (5.19.0-23.24) ...                                                                               
/etc/kernel/header_postinst.d/dkms:                                                                                                         
 * dkms: running auto installation service for kernel 5.19.0-23-generic                                                                     
   ...done. 
  1. dkms status → returns →
nvidia/520.56.06, 5.19.0-23-generic, x86_64: installed (WARNING! Diff between built and installed module!)                                  
nvidia/520.56.06, 6.0.5-060005-generic, x86_64: installed                                                                                   
openrazer-driver/3.4.0, 5.19.0-23-generic, x86_64: installed                                                                                
openrazer-driver/3.4.0, 6.0.5-060005-generic, x86_64: installed 
  1. sudo dkms remove nvidia/520.56.06
Module nvidia-520.56.06 for kernel 5.19.0-23-generic (x86_64).                                                                              
Before uninstall, this module version was ACTIVE on this kernel.                                                                            
                                                                                                                                            
nvidia.ko:                                                                                                                                  
 - Uninstallation                                                                                                                           
   - Deleting from: /lib/modules/5.19.0-23-generic/                                                                                         
rmdir: failed to remove '': No such file or directory                                                                                       
 - Original module                                                                                                                          
   - No original module was found for this module on this kernel.                                                                           
   - Use the dkms install command to reinstall any previous module version.                                                                 
                                                                                                                                            
nvidia-modeset.ko:                                                                                                                          
 - Uninstallation                                                                                                                           
   - Deleting from: /lib/modules/5.19.0-23-generic/updates/dkms/                                                                            
 - Original module                                                                                                                          
   - No original module was found for this module on this kernel.                                                                           
   - Use the dkms install command to reinstall any previous module version.

nvidia-drm.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-23-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

nvidia-uvm.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-23-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

nvidia-peermem.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-23-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.
depmod...
  1. sudo dkms install nvidia/520.56.06
Sign command: /usr/lib/linux-kbuild-5.19/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Binary kmod-sign not found, modules won't be signed

Building module:
Cleaning build area...
unset ARCH; [ ! -h /usr/bin/cc ] && export CC=/usr/bin/gcc; env NV_VERBOSE=1 'make' -j8 NV_EXCLUDE_BUILD_MODULES='' KERNEL_UNAME=5.19.0-23-generic IGNORE_XEN_PRESENCE=1 IGNORE_CC_MISMATCH=1 SYSSRC=/lib/modules/5.19.0-23-generic/build LD=/usr/bin/ld.bfd CONFIG_X86_KERNEL_IBT= modules..........
Cleaning build area...

nvidia.ko:
Running module version sanity check.
Error! Module version 520.56.06 for nvidia.ko
is not newer than what is already found in kernel 5.19.0-23-generic (520.56.06).
You may override by specifying --force.

nvidia-modeset.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/

nvidia-drm.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/

nvidia-uvm.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/

nvidia-peermem.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/
depmod...
  1. sudo modprobe nvidia
modprobe: ERROR: could not insert 'nvidia': Exec format error

Please use --force as specified
sudo dkms install nvidia/520.56.06 --force

@generix,

  1. sudo dkms remove --force nvidia/520.56.06
Module nvidia-520.56.06 for kernel 5.19.0-23-generic (x86_64).                                                                              
Before uninstall, this module version was ACTIVE on this kernel.                                                                            
                                                                                                                                            
nvidia.ko:                                                                                                                                  
 - Uninstallation                                                                                                                           
   - Deleting from: /lib/modules/5.19.0-23-generic/                                                                                         
rmdir: failed to remove '': No such file or directory                                                                                       
 - Original module                                                                                                                          
   - No original module was found for this module on this kernel.                                                                           
   - Use the dkms install command to reinstall any previous module version.                                                                 
                                                                                                                                            
nvidia-modeset.ko:                                                                                                                          
 - Uninstallation                                                                                                                           
   - Deleting from: /lib/modules/5.19.0-23-generic/updates/dkms/                                                                            
 - Original module                                                                                                                          
   - No original module was found for this module on this kernel.                                                                           
   - Use the dkms install command to reinstall any previous module version.

nvidia-drm.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-23-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

nvidia-uvm.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-23-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

nvidia-peermem.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-23-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.
depmod...
  1. dkms status
nvidia/520.56.06, 6.0.5-060005-generic, x86_64: installed
openrazer-driver/3.4.0, 5.19.0-23-generic, x86_64: installed
openrazer-driver/3.4.0, 6.0.5-060005-generic, x86_64: installed
  1. sudo dkms install --force nvidia/520.56.06
Sign command: /usr/lib/linux-kbuild-5.19/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub
Binary kmod-sign not found, modules won't be signed

Building module:
Cleaning build area...
unset ARCH; [ ! -h /usr/bin/cc ] && export CC=/usr/bin/gcc; env NV_VERBOSE=1 'make' -j8 NV_EXCLUDE_BUILD_MODULES='' KERNEL_UNAME=5.19.0-23-generic IGNORE_XEN_PRESENCE=1 IGNORE_CC_MISMATCH=1 SYSSRC=/lib/modules/5.19.0-23-generic/build LD=/usr/bin/ld.bfd CONFIG_X86_KERNEL_IBT= modules..........
Cleaning build area...

nvidia.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/

nvidia-modeset.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/

nvidia-drm.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/

nvidia-uvm.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/

nvidia-peermem.ko:
Running module version sanity check.
 - Original module
 - Installation
   - Installing to /lib/modules/5.19.0-23-generic/updates/dkms/
depmod...
  1. sudo modprobe nvidia
modprobe: ERROR: could not insert 'nvidia': Exec format error
  1. dkms status
nvidia/520.56.06, 5.19.0-23-generic, x86_64: installed
nvidia/520.56.06, 6.0.5-060005-generic, x86_64: installed
openrazer-driver/3.4.0, 5.19.0-23-generic, x86_64: installed
openrazer-driver/3.4.0, 6.0.5-060005-generic, x86_64: installed

Looks like the reinstall still didnt work?

Your build system is broken. Please post the output of
cc -v
ld -v

@generix, see below

➜  ~ cc -v
Using built-in specs.
COLLECT_GCC=cc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/12/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 12.2.0-3ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-12/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-12 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-12-U8K4Qv/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-U8K4Qv/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-offload-defaulted --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.2.0 (Ubuntu 12.2.0-3ubuntu1) 
➜  ~ ld -v
GNU ld (GNU Binutils for Ubuntu) 2.39

I ran a history | grep 'gcc' → which outputted sudo apt install gcc-8. Maybe when I ran this command I broke the build system? Whats the fix for that?

Looks fine. Please try purging and installing the kernel headers instead of reinstalling them.

OK, @generix I did the following, still cannot modprobe nivida…

➜  ~ sudo apt remove --purge linux-headers-$(uname -r)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  linux-headers-5.19.0-23
Use 'sudo apt autoremove' to remove it.
The following packages will be REMOVED:
  linux-headers-5.19.0-23-generic* linux-headers-generic*
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 27.6 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 523239 files and directories currently installed.)
Removing linux-headers-generic (5.19.0.23.22) ...
Removing linux-headers-5.19.0-23-generic (5.19.0-23.24) ...
➜  ~ sudo apt install linux-headers-$(uname -r)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  linux-headers-5.19.0-23-generic
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,270 kB of archives.
After this operation, 27.6 MB of additional disk space will be used.
Get:1 http://ca.archive.ubuntu.com/ubuntu kinetic-updates/main amd64 linux-headers-5.19.0-23-generic amd64 5.19.0-23.24 [3,270 kB]
Fetched 3,270 kB in 0s (11.6 MB/s)                        
Selecting previously unselected package linux-headers-5.19.0-23-generic.
(Reading database ... 513426 files and directories currently installed.)
Preparing to unpack .../linux-headers-5.19.0-23-generic_5.19.0-23.24_amd64.deb ...
Unpacking linux-headers-5.19.0-23-generic (5.19.0-23.24) ...
Setting up linux-headers-5.19.0-23-generic (5.19.0-23.24) ...
/etc/kernel/header_postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.19.0-23-generic
   ...done.
➜  ~ sudo modprobe nvidia
modprobe: ERROR: could not insert 'nvidia': Exec format error
➜  ~ dkms status
nvidia/520.56.06, 5.19.0-23-generic, x86_64: installed
nvidia/520.56.06, 6.0.5-060005-generic, x86_64: installed
openrazer-driver/3.4.0, 5.19.0-23-generic, x86_64: installed
openrazer-driver/3.4.0, 6.0.5-060005-generic, x86_64: installed
1 Like

Please post the output of
modinfo nvidia

@generix,

filename:       /lib/modules/5.19.0-23-generic/updates/dkms/nvidia.ko
firmware:       nvidia/520.56.06/gsp.bin
alias:          char-major-195-*
version:        520.56.06
supported:      external
license:        NVIDIA
srcversion:     16DD2AB6F8E16BF9E137E59
alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
depends:        drm
retpoline:      Y
name:           nvidia
vermagic:       5.19.0-23-generic SMP preempt mod_unload modversions 
parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
parm:           NVreg_ResmanDebugLevel:int
parm:           NVreg_RmLogonRC:int
parm:           NVreg_ModifyDeviceFiles:int
parm:           NVreg_DeviceFileUID:int
parm:           NVreg_DeviceFileGID:int
parm:           NVreg_DeviceFileMode:int
parm:           NVreg_InitializeSystemMemoryAllocations:int
parm:           NVreg_UsePageAttributeTable:int
parm:           NVreg_EnablePCIeGen3:int
parm:           NVreg_EnableMSI:int
parm:           NVreg_TCEBypassMode:int
parm:           NVreg_EnableStreamMemOPs:int
parm:           NVreg_RestrictProfilingToAdminUsers:int
parm:           NVreg_PreserveVideoMemoryAllocations:int
parm:           NVreg_EnableS0ixPowerManagement:int
parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
parm:           NVreg_DynamicPowerManagement:int
parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
parm:           NVreg_EnableGpuFirmware:int
parm:           NVreg_EnableGpuFirmwareLogs:int
parm:           NVreg_OpenRmEnableUnsupportedGpus:int
parm:           NVreg_EnableUserNUMAManagement:int
parm:           NVreg_MemoryPoolSize:int
parm:           NVreg_KMallocHeapMaxSize:int
parm:           NVreg_VMallocHeapMaxSize:int
parm:           NVreg_IgnoreMMIOCheck:int
parm:           NVreg_NvLinkDisable:int
parm:           NVreg_EnablePCIERelaxedOrderingMode:int
parm:           NVreg_RegisterPCIDriver:int
parm:           NVreg_EnableDbgBreakpoint:int
parm:           NVreg_RegistryDwords:charp
parm:           NVreg_RegistryDwordsPerDevice:charp
parm:           NVreg_RmMsg:charp
parm:           NVreg_GpuBlacklist:charp
parm:           NVreg_TemporaryFilePath:charp
parm:           NVreg_ExcludedGpus:charp
parm:           NVreg_DmaRemapPeerMmio:int
parm:           rm_firmware_active:charp

Update. I uninstalled nvidia, purged and reinstalled my headers, and rebooted my pc. I just ran an install of the 520 driver, and now I get this with modprobe:

sudo modprobe nvidia
[sudo] password for cboin: 
modprobe: ERROR: could not insert 'nvidia': No such device

This is a different error from before. @generix, what do you think? Also do you have patreon? You contribute quite a bit to the community, I wanted to donate.

OKOKOKOKO I got it working.

Thank you so much for your help @generix.

Heres what I did, line by line.

 sudo apt install --reinstall linux-image-generic
 sudo apt install --reinstall linux-headers-generic

Next,

        sudo apt remove --purge '^nvidia-.*'
        sudo apt remove --purge '^libnvidia-.*'
        sudo rm /etc/X11/xorg.conf | true
        sudo rm /etc/X11/xorg.conf.d/90-nvidia-primary.conf | true
        sudo rm /usr/share/X11/xorg.conf.d/10-nvidia.conf | true
        sudo rm /usr/share/X11/xorg.conf.d/11-nvidia-prime.conf | true
        sudo rm /etc/modprobe.d/nvidia-kms.conf | true
        sudo rm /lib/modprobe.d/nvidia-kms.conf | true
        sudo apt update -y && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean -y && sudo apt autoclean -y

Rebooted.

Installed nvidia-driver-520 from the additional drivers tab of ubuntu.

Rebooted

nvidia-smi
Thu Oct 27 18:42:58 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0  On |                  N/A |
| 33%   38C    P0    32W / 225W |    826MiB /  8192MiB |     17%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2389      G   /usr/lib/xorg/Xorg                214MiB |
|    0   N/A  N/A      2713      G   /usr/bin/gnome-shell              195MiB |
|    0   N/A  N/A      3024      G   ...3/usr/lib/firefox/firefox      237MiB |
|    0   N/A  N/A      3036      G   ...b/thunderbird/thunderbird      145MiB |
|    0   N/A  N/A      3225      G   ...mail/bridge/proton-bridge        2MiB |
|    0   N/A  N/A      3567      G   ...AAAAAAAAA= --shared-files       26MiB |
+-----------------------------------------------------------------------------+

Works! Once again, if you have patreon I would be happy to donate.

8 Likes

@christianboin this post was a lifesaver. I too was able to get it working again after following your outline. It appears that the Ubuntu 20.04 install seems to pick the 520 open-kernel meta package, which does not work for me. The plain 520 meta package does work. Thanks for this detailed post!

5 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.