Tesla M40 24GB not recognized

Hello :)

I have a Tesla M40 24GB GPU, which is not recognized by the nvidia-settings and nvidia-smi.
(i7-4790, 4x4GB DDR3 1600MHz, Nvidia Tesla M40 24GB, Asus Maximus VII Ranger, 512GB M.2 SSD, Above 4G Decoding activated, Secure Boot deactivated)

$ nvidia-settings:

ERROR: A supplied argument is invalid


(nvidia-settings:8043): GLib-GObject-CRITICAL **: 21:02:06.219: g_object_unref: assertion 'G_IS_OBJECT (object)' failed

** (nvidia-settings:8043): CRITICAL **: 21:02:06.220: ctk_powermode_new: assertion '(ctrl_target != NULL) && (ctrl_target->h != NULL)' failed
** Message: 21:02:06.250: PRIME: Requires offloading
** Message: 21:02:06.250: PRIME: is it supported? yes
** Message: 21:02:06.277: PRIME: Usage: /usr/bin/prime-select nvidia|intel|on-demand|query
** Message: 21:02:06.277: PRIME: on-demand mode: "1"
** Message: 21:02:06.277: PRIME: is "on-demand" mode supported? yes

$ nvidia-smi:

No devices were found

Linux however does detect the M40 and the newest drivers:

$ neofetch:

            .-/+oossssoo+/-.               ion-san@Homestation 
        `:+ssssssssssssssssss+:`           ------------------- 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 22.04.3 LTS x86_64 
    .ossssssssssssssssssdMMMNysssso.       Kernel: 6.2.0-33-generic 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Uptime: 46 mins 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Packages: 1731 (dpkg), 11 (snap) 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Shell: bash 5.1.16 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Resolution: 1920x1080 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   DE: GNOME 42.9 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   WM: Mutter 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   WM Theme: Adwaita 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Theme: Yaru [GTK2/3] 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Icons: Yaru [GTK2/3] 
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/    Terminal: gnome-terminal 
  +sssssssssdmydMMMMMMMMddddyssssssss+     CPU: Intel i7-4790 (8) @ 4.000GHz 
   /ssssssssssshdmNNNNmyNMMMMhssssss/      GPU: Intel HD Graphics 
    .ossssssssssssssssssdMMMNysssso.       GPU: NVIDIA Tesla M40 
      -+sssssssssssssssssyyyssss+-         Memory: 2522MiB / 15860MiB 
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.     

$ inxi -G

Graphics:
  Device-1: Intel Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics
    driver: i915 v: kernel
  Device-2: NVIDIA GM200GL [Tesla M40] driver: nvidia v: 535.86.05
  Display: x11 server: X.Org v: 1.21.1.4 driver: X: loaded: modesetting
    gpu: i915 resolution: 1920x1080~60Hz
  OpenGL: renderer: Mesa Intel HD Graphics 4600 (HSW GT2)
    v: 4.6 Mesa 23.0.4-0ubuntu1~22.04.1

$ lspci

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
00:1f.0 ISA bridge: Intel Corporation Z97 Chipset LPC Controller
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
01:00.0 3D controller: NVIDIA Corporation GM200GL [Tesla M40] (rev a1)
03:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. Device 5013 (rev 01)

$ sudo lshw -c video

  *-display                 
       description: 3D controller
       product: GM200GL [Tesla M40]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress bus_master cap_list
       configuration: driver=nvidia latency=0
       resources: irq:16 memory:f0000000-f0ffffff
  *-display
       description: VGA compatible controller
       product: Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
       vendor: Intel Corporation
       physical id: 2
       bus info: pci@0000:00:02.0
       logical name: /dev/fb0
       version: 06
       width: 64 bits
       clock: 33MHz
       capabilities: msi pm vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=i915 latency=0 resolution=1920,1080
       resources: irq:39 memory:f7800000-f7bfffff memory:e0000000-efffffff ioport:f000(size=64) memory:c0000-dffff

$ dpkg -l | grep nvidia-

ii  libnvidia-cfg1-535:amd64                   535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-535                       535.86.05-0ubuntu0.22.04.1              all          Shared files used by the NVIDIA libraries
ii  libnvidia-compute-535:amd64                535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA libcompute package
ii  libnvidia-compute-535:i386                 535.86.05-0ubuntu0.22.04.1              i386         NVIDIA libcompute package
ii  libnvidia-decode-535:amd64                 535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-decode-535:i386                  535.86.05-0ubuntu0.22.04.1              i386         NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-535:amd64                 535.86.05-0ubuntu0.22.04.1              amd64        NVENC Video Encoding runtime library
ii  libnvidia-encode-535:i386                  535.86.05-0ubuntu0.22.04.1              i386         NVENC Video Encoding runtime library
ii  libnvidia-extra-535:amd64                  535.86.05-0ubuntu0.22.04.1              amd64        Extra libraries for the NVIDIA driver
ii  libnvidia-fbc1-535:amd64                   535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-fbc1-535:i386                    535.86.05-0ubuntu0.22.04.1              i386         NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-535:amd64                     535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-gl-535:i386                      535.86.05-0ubuntu0.22.04.1              i386         NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  linux-modules-nvidia-535-6.2.0-26-generic  6.2.0-26.26~22.04.1+2                   amd64        Linux kernel nvidia modules for version 6.2.0-26
ii  linux-modules-nvidia-535-6.2.0-33-generic  6.2.0-33.33~22.04.1                     amd64        Linux kernel nvidia modules for version 6.2.0-33
ii  linux-modules-nvidia-535-generic-hwe-22.04 6.2.0-33.33~22.04.1                     amd64        Extra drivers for nvidia-535 for the generic-hwe-22.04 flavour
ii  linux-objects-nvidia-535-6.2.0-26-generic  6.2.0-26.26~22.04.1+2                   amd64        Linux kernel nvidia modules for version 6.2.0-26 (objects)
ii  linux-objects-nvidia-535-6.2.0-33-generic  6.2.0-33.33~22.04.1                     amd64        Linux kernel nvidia modules for version 6.2.0-33 (objects)
ii  linux-signatures-nvidia-6.2.0-26-generic   6.2.0-26.26~22.04.1+2                   amd64        Linux kernel signatures for nvidia modules for version 6.2.0-26-generic
ii  linux-signatures-nvidia-6.2.0-33-generic   6.2.0-33.33~22.04.1                     amd64        Linux kernel signatures for nvidia modules for version 6.2.0-33-generic
ii  nvidia-compute-utils-535                   535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA compute utilities
ii  nvidia-driver-535                          535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA driver metapackage
ii  nvidia-firmware-535-535.86.05              535.86.05-0ubuntu0.22.04.1              amd64        Firmware files used by the kernel module
ii  nvidia-kernel-common-535                   535.86.05-0ubuntu0.22.04.1              amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-535                   535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA kernel source package
ii  nvidia-prime                               0.8.17.1                                all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                            510.47.03-0ubuntu1                      amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-535                           535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA driver support binaries
ii  screen-resolution-extra                    0.18.2                                  all          Extension for the nvidia-settings control panel
ii  xserver-xorg-video-nvidia-535              535.86.05-0ubuntu0.22.04.1              amd64        NVIDIA binary Xorg driver

$ dmesg | grep nvidia

[    1.760496] audit: type=1400 audit(1695497467.785:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=395 comm="apparmor_parser"
[    1.760499] audit: type=1400 audit(1695497467.785:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=395 comm="apparmor_parser"
[    1.879892] nvidia: loading out-of-tree module taints kernel.
[    1.879900] nvidia: module license 'NVIDIA' taints kernel.
[    2.090346] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
[    2.091282] nvidia 0000:01:00.0: enabling device (0000 -> 0002)
[    2.227245] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  535.86.05  Fri Jul 14 20:20:58 UTC 2023
[    2.258338] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[    2.273607] [drm:nv_drm_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Failed to allocate NvKmsKapiDevice
[    2.273844] [drm:nv_drm_probe_devices [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Failed to register device
[    2.471412] nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
[    2.504381] nvidia-uvm: Loaded the UVM driver, major device number 508.
[  119.878983] audit: type=1400 audit(1695497585.406:64): apparmor="DENIED" operation="open" class="file" profile="snap.firefox.firefox" name="/home/ion-san/nvidia-bug-report.log.gz" pid=2381 comm=4261636B67726F7E6F6F6C20233133 requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[  124.706280] audit: type=1400 audit(1695497590.234:68): apparmor="DENIED" operation="open" class="file" profile="snap.firefox.firefox" name="/home/ion-san/.nvidia-settings-rc" pid=2381 comm="pool-firefox" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[  128.014718] audit: type=1400 audit(1695497593.542:71): apparmor="DENIED" operation="open" class="file" profile="snap.firefox.firefox" name="/home/ion-san/nvidia-bug-report.log.gz" pid=2381 comm=4261636B67726F7E6F6F6C20233133 requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

Screenshot from 2023-09-23 21-23-11
nvidia-bug-report.log.gz (140.0 KB)

Please help, I have already tried re-installing the drivers, altering the xorg.config, blacklisting the nouveau drivers, switching between the prime-select modes and re-installing ubuntu 22.04 LTS (had the same problems on Manjaro). Now I’m quite dazed and confused and would appreciate some advice (thanks in advance :)