Cannot install NVIDIA driver for Tesla T4

I am trying to install NVIDIA driver on Ubuntu 20.04. However, I am getting an error

ERROR: Unable to load the kernel module 'nvidia.ko'.  This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or i
f another driver, such as nouveau, is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA device(s), or no NVIDIA device installed in this system is supported by this NVIDIA Linux graphics driver release.

Please see the log entries 'Kernel module load error' and 'Kernel messages' at the end of the file '/var/log/nvidia-installer.log' for more information.
-> Kernel module load error: No such device
-> Kernel messages:
[ 1716.519323] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=159.65.178.29 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=44827 PROTO=TCP SPT=61953 DPT=6156 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1734.232201] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=192.241.215.18 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=56758 DPT=111 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1785.312080] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=91.240.118.224 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=64933 PROTO=TCP SPT=59458 DPT=4489 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1808.290152] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=45.61.185.149 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=54321 PROTO=TCP SPT=52998 DPT=81 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1813.229957] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=221.15.186.127 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=26980 PROTO=TCP SPT=16269 DPT=23 WINDOW=14212 RES=0x00 SYN URGP=0 
[ 1845.598186] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=198.235.24.185 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=251 ID=54321 PROTO=TCP SPT=57303 DPT=8991 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1858.506370] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=192.241.216.116 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=39506 DPT=2376 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1870.199631] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=223.71.167.165 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=105 ID=17098 PROTO=TCP SPT=50014 DPT=1260 WINDOW=29200 RES=0x00 SYN URGP=0 
[ 1877.011559] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=193.163.125.160 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=242 ID=38606 PROTO=TCP SPT=38244 DPT=21295 WINDOW=14600 RES=0x00 SYN URGP=0 
[ 1893.130734] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=89.248.163.189 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=52512 PROTO=TCP SPT=52296 DPT=808 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1917.645383] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=79.124.62.130 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=34204 PROTO=TCP SPT=40379 DPT=19194 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1944.118112] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=167.94.146.26 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=43 ID=34170 PROTO=TCP SPT=12836 DPT=31276 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1973.714434] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=192.241.208.79 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=58644 DPT=113 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1987.401601] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=186.232.197.29 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=240 ID=49131 PROTO=TCP SPT=45220 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1996.697671] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=109.247.140.27 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=18089 PROTO=TCP SPT=64859 DPT=23 WINDOW=33579 RES=0x00 SYN URGP=0 
[ 2025.436208] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=91.240.118.77 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=12531 PROTO=TCP SPT=59236 DPT=8365 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 2039.344576] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=79.124.62.86 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=5485 PROTO=TCP SPT=40378 DPT=13 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 2077.200893] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 2077.214076] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
[ 2077.214083] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
               NVRM: BAR1 is 0M @ 0x0 (PCI:0000:01:00.0)
[ 2077.232039] nvidia: probe of 0000:01:00.0 failed with error -1
[ 2077.232160] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 2077.232165] NVRM: None of the NVIDIA devices were initialized.
[ 2077.232948] nvidia-nvlink: Unregistered Nvlink Core, major device number 511
ERROR: Installation has failed.  Please see the file '/var/log/nvidia-installer.log' for details.  You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.

@tbezdan hello, are you still having trouble?

Can you post more information about the commands you used to install the driver?

Related, there is a resource on NGC that installs the necessary drivers and Docker on a clean Ubuntu 18.04/20.04/22.04 machine. It’s been tested on a fresh OS install, it may also work on an existing machine where other packages/drivers have been installed.
https://ngc.nvidia.com/resources/ea-reopt-member-zone:setup_ubuntu_for_cuopt

@tbezdan also note the new cuOpt-Resources repository.

There are scripts there under “cloud-scripts” that will create a VM for you on AWS, Azure, or GCP running cuOpt on top of Cloud Native Core (single node Kubernetes), or you can install the same stack locally on an Ubuntu 22.04 machine that you provide yourself. This may be useful to you.

Best watch display case

Thanks for the scripts