HP-Z4-G4-Workstation can not switch from Nouveau driver to Nvidia driver on ubuntu 18.04

I select nvidia driver from software & updates, then apply changes, it goes back to Nouveau.
my nvidia hardware is GP104GL Quadro P4000. I get black display after wake-up from suspend. tried nvidia setting
ERROR: NVIDIA driver is not loaded

ERROR: Unable to load info from any available system

(nvidia-settings:34365): GLib-GObject-CRITICAL **: 14:39:14.046: g_object_unref: assertion ‘G_IS_OBJECT (object)’ failed
** Message: 14:39:14.051: PRIME: No offloading required. Abort
** Message: 14:39:14.051: PRIME: is it supported? no
nvidia-bug-report.log.gz (108.0 KB)

The kernel driver is missing. Please post the output of
dkms status

Nothing present

Please post the output of
dpkg -l |grep nvidia

ii libnvidia-cfg1-435:amd64 435.21-0ubuntu0.18.04.2 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-435 435.21-0ubuntu0.18.04.2 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-435:amd64 435.21-0ubuntu0.18.04.2 amd64 NVIDIA libcompute package
ii libnvidia-compute-435:i386 435.21-0ubuntu0.18.04.2 i386 NVIDIA libcompute package
ii libnvidia-decode-435:amd64 435.21-0ubuntu0.18.04.2 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-decode-435:i386 435.21-0ubuntu0.18.04.2 i386 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-435:amd64 435.21-0ubuntu0.18.04.2 amd64 NVENC Video Encoding runtime library
ii libnvidia-encode-435:i386 435.21-0ubuntu0.18.04.2 i386 NVENC Video Encoding runtime library
ii libnvidia-fbc1-435:amd64 435.21-0ubuntu0.18.04.2 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-fbc1-435:i386 435.21-0ubuntu0.18.04.2 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-435:amd64 435.21-0ubuntu0.18.04.2 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-gl-435:i386 435.21-0ubuntu0.18.04.2 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-ifr1-435:amd64 435.21-0ubuntu0.18.04.2 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime library
ii libnvidia-ifr1-435:i386 435.21-0ubuntu0.18.04.2 i386 NVIDIA OpenGL-based Inband Frame Readback runtime library
ii nvidia-compute-utils-435 435.21-0ubuntu0.18.04.2 amd64 NVIDIA compute utilities
ii nvidia-dkms-435 435.21-0ubuntu0.18.04.2 amd64 NVIDIA DKMS package
ii nvidia-driver-435 435.21-0ubuntu0.18.04.2 amd64 NVIDIA driver metapackage
ii nvidia-kernel-common-435 435.21-0ubuntu0.18.04.2 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-435 435.21-0ubuntu0.18.04.2 amd64 NVIDIA kernel source package
ii nvidia-prime 0.8.16~0.18.04.1 all Tools to enable NVIDIA’s Prime
ii nvidia-settings 470.57.01-0ubuntu0.18.04.1 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-435 435.21-0ubuntu0.18.04.2 amd64 NVIDIA driver support binaries
ii xserver-xorg-video-nvidia-435 435.21-0ubuntu0.18.04.2 amd64 NVIDIA binary Xorg driver

Please try running
sudo dkms install nvidia/435.21
afterwards, post the output of
dkms status

kai@kai-HP-Z4-G4-Workstation:~/Projects$ sudo dkms install nvidia/435.21

Creating symlink /var/lib/dkms/nvidia/435.21/source →
/usr/src/nvidia-435.21

DKMS: add completed.

Kernel preparation unnecessary for this kernel. Skipping…

Building module:
cleaning build area…
unset ARCH; env NV_VERBOSE=1 ‘make’ -j16 NV_EXCLUDE_BUILD_MODULES=‘’ KERNEL_UNAME=5.4.0-91-generic IGNORE_XEN_PRESENCE=1 IGNORE_CC_MISMATCH=1 SYSSRC=/lib/modules/5.4.0-91-generic/build LD=/usr/bin/ld.bfd modules…(bad exit status: 2)
Error! Bad return status for module build on kernel: 5.4.0-91-generic (x86_64)
Consult /var/lib/dkms/nvidia/435.21/build/make.log for more information.

kai@kai-HP-Z4-G4-Workstation:~/Projects$ dkms status
nvidia, 435.21: added

Please attach the make log
/var/lib/dkms/nvidia/435.21/build/make.log

make.log (801.4 KB)

The driver version is too old for the currently running kernel. Are there any newer versions available in Software&Updates?

the latest one is 495, but no one works

The bug report shows the OP already tried to install v. 460, 470 and 495. Something went wrong in the process.
So I suggest purge all what is there first:
sudo apt purge nvidia* libnvidia*
and then
sudo apt install nvidia-driver-470

kai@kai-HP-Z4-G4-Workstation:~/Projects$ sudo apt install nvidia-driver-470
Reading package lists… Done
Building dependency tree
Reading state information… Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies.
nvidia-driver-470 : Depends: libnvidia-decode-470 (= 470.86-0ubuntu0.18.04.1) but it is not going to be installed
Depends: libnvidia-encode-470 (= 470.86-0ubuntu0.18.04.1) but it is not going to be installed
Depends: libnvidia-fbc1-470 (= 470.86-0ubuntu0.18.04.1) but it is not going to be installed
Recommends: libnvidia-compute-470:i386 (= 470.86-0ubuntu0.18.04.1)
Recommends: libnvidia-decode-470:i386 (= 470.86-0ubuntu0.18.04.1)
Recommends: libnvidia-encode-470:i386 (= 470.86-0ubuntu0.18.04.1)
Recommends: libnvidia-fbc1-470:i386 (= 470.86-0ubuntu0.18.04.1)
E: Unable to correct problems, you have held broken packages.

I guess you have some package installed which depends on an older version of the driver. Please post the output of
dpkg -l |grep nvidia
and dpkg -l |grep cuda

kai@kai-HP-Z4-G4-Workstation:~$ dpkg -l |grep nvidia
ii libnvidia-compute-435:i386 435.21-0ubuntu0.18.04.2 i386 NVIDIA libcompute package
ii libnvidia-decode-435:i386 435.21-0ubuntu0.18.04.2 i386 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-435:i386 435.21-0ubuntu0.18.04.2 i386 NVENC Video Encoding runtime library
ii libnvidia-fbc1-435:i386 435.21-0ubuntu0.18.04.2 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
kai@kai-HP-Z4-G4-Workstation:~$ dpkg -l |grep cuda
kai@kai-HP-Z4-G4-Workstation:~$

Do you get any errors when you re-run
sudo apt remove libnvidia*
and are those packages gone afterwards?

no error, this time I run sudo apt remove libnvidia* followed by sudo apt autoremove, I can switch to nvidia 495, it works now. Thanks!