Cannot install NVIDIA driver for Tesla T4

Accelerated Computing CUDA CUDA Setup and Installation
, ,
1

I am trying to install NVIDIA driver on Ubuntu 20.04. However, I am getting an error

ERROR: Unable to load the kernel module 'nvidia.ko'.  This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or i
f another driver, such as nouveau, is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA device(s), or no NVIDIA device installed in this system is supported by this NVIDIA Linux graphics driver release.

Please see the log entries 'Kernel module load error' and 'Kernel messages' at the end of the file '/var/log/nvidia-installer.log' for more information.
-> Kernel module load error: No such device
-> Kernel messages:
[ 1716.519323] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=159.65.178.29 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=44827 PROTO=TCP SPT=61953 DPT=6156 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1734.232201] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=192.241.215.18 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=56758 DPT=111 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1785.312080] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=91.240.118.224 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=64933 PROTO=TCP SPT=59458 DPT=4489 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1808.290152] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=45.61.185.149 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=54321 PROTO=TCP SPT=52998 DPT=81 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1813.229957] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=221.15.186.127 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=26980 PROTO=TCP SPT=16269 DPT=23 WINDOW=14212 RES=0x00 SYN URGP=0 
[ 1845.598186] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=198.235.24.185 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=251 ID=54321 PROTO=TCP SPT=57303 DPT=8991 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1858.506370] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=192.241.216.116 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=39506 DPT=2376 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1870.199631] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=223.71.167.165 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=105 ID=17098 PROTO=TCP SPT=50014 DPT=1260 WINDOW=29200 RES=0x00 SYN URGP=0 
[ 1877.011559] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=193.163.125.160 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=242 ID=38606 PROTO=TCP SPT=38244 DPT=21295 WINDOW=14600 RES=0x00 SYN URGP=0 
[ 1893.130734] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=89.248.163.189 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=52512 PROTO=TCP SPT=52296 DPT=808 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1917.645383] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=79.124.62.130 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=34204 PROTO=TCP SPT=40379 DPT=19194 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1944.118112] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=167.94.146.26 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=43 ID=34170 PROTO=TCP SPT=12836 DPT=31276 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1973.714434] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=192.241.208.79 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=58644 DPT=113 WINDOW=65535 RES=0x00 SYN URGP=0 
[ 1987.401601] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=186.232.197.29 DST=89.216.51.58 LEN=44 TOS=0x00 PREC=0x00 TTL=240 ID=49131 PROTO=TCP SPT=45220 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 1996.697671] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=109.247.140.27 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=50 ID=18089 PROTO=TCP SPT=64859 DPT=23 WINDOW=33579 RES=0x00 SYN URGP=0 
[ 2025.436208] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=91.240.118.77 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=12531 PROTO=TCP SPT=59236 DPT=8365 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 2039.344576] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=79.124.62.86 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=5485 PROTO=TCP SPT=40378 DPT=13 WINDOW=1024 RES=0x00 SYN URGP=0 
[ 2077.200893] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 2077.214076] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
[ 2077.214083] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
               NVRM: BAR1 is 0M @ 0x0 (PCI:0000:01:00.0)
[ 2077.232039] nvidia: probe of 0000:01:00.0 failed with error -1
[ 2077.232160] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 2077.232165] NVRM: None of the NVIDIA devices were initialized.
[ 2077.232948] nvidia-nvlink: Unregistered Nvlink Core, major device number 511
ERROR: Installation has failed.  Please see the file '/var/log/nvidia-installer.log' for details.  You may find suggestions on fixing installation problems in the README available on the Linux driver download page at www.nvidia.com.
2 
[ 2077.214083] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
               NVRM: BAR1 is 0M @ 0x0 (PCI:0000:01:00.0)
[ 2077.232039] nvidia: probe of 0000:01:00.0 failed with error -1

This seems to be the culprit. The card does not have any space assigned in the base address register.

Also see here: driver install failed with error "NVRM: BAR1 is 0M" on GA100, ubuntu1804/1604

3

Thank you very much for your reply! In the provided link, there is no solution for this issue. I would appreciate it if you could give more precise instructions for the solution.

4

Tesla cards often have significantly larger than normal BAR size requirements and so may need the appropriate BIOS settings in order to achieve this. This is one of the reasons these cards are sold fitted to Nvidia certified systems.

See Roberts reply, (second post): Plugging Tesla K80 results in PCI resource allocation error

Also, certified systems, here and here.

5

Thank you @rs277 , I am new to Ubuntu, could you provide an example of how to do the BIOS settings for the BAR size by using ssh?

6

Sorry, I’ve never needed to do this and have never owned a PC that would enable it to be done. Google may help.

7

The problem is that every board manufacturer may have different settings in the BIOS. And some lower end mainboard don’t offer the required setting at all.

  1. Updating the BIOS to the latest published version from the vendor may help.
  2. Try enabling “Above 4G decoding” in the BIOS, which may appear under PCI subsystem settings. Some higher end, and most workstation and server mainboards should have this.
8

It looks like the installation was successful now, but when I run nvidia-smi, I get the following: Failed to initialize NVML: Driver/library version mismatch

I tried to remove and uninstall nvidia, then to install it again, but I am still getting the same.

9

Does the error go away after a reboot? Usually that is a sign of the nvidia driver binary not matching the loaded kernel module.

I have seen this error occur on forced driver security updates in Ubuntu that were installed by the automatic update mechanism. I would get this error until I finally rebooted.

If the error persists, try to find out from which package the current nvidia-smi binary was installed and see if that matches the loaded nvidia driver and kernel module.

10

Yes, I rebooted it. And checked the versions.

NVRM: this kernel module has the version 515.65.01. Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
[ 813.779957] [UFW BLOCK] IN=eno1 OUT= MAC=18:66:da:20:86:2f:b8:69:f4:01:56:96:08:00 SRC=104.156.155.2 DST=89.216.51.58 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=37369 PROTO=TCP SPT=41076 DPT=17000 WINDOW=1024 RES=0x00 SYN URGP=0