NVRM: This PCI I/O region assigned to your NVIDIA device is invalid

Something must be triggering nvidia.ko to try to load. Can you try to boot into single-user rescue mode or something? Alternatively, you could temporarily uninstall or blacklist the nvidia driver for this experiment so that it can’t load while you’re trying to check the kernel logs.

Thanks, @aplattner. For the life of me, I can’t get the messages to stop. Tried booting in recovery mode. And tried blacklisting in modprobe.d. If I ‘apt remove’ all nvidia packages, will nvidia-bug-report.sh still work, and give useful information? Will dmesg?

Also, is there a way to clear whatever file/buffer/log is drawn upon to create the nvidia-bug-report.log? The generated .log file has info that spans several days and is getting to be gigantic in size.

Might be nvidia-persistenced.
Nevermind, just uninstall the driver by running (in an empty directory)
sudo apt remove nvidia*
afterwards, just create the dmesg log.

Thanks, @generix. I removed the nvidia packages. Log file is attached.dmesg.log (70.4 KB)

Looks like a bios boot. Please disable CSM in bios and do a clean EFI boot reinstall. Afterwards, don’t install the driver but provide a new dmesg log.

That’s pushing me past my experience level. With CSM disabled, I can’t boot from the drive (perhaps that was expected). Can you point me toward some resources/info on how to “do a clean EFI boot reinstall?”

You’ll have to format and reinstall, i.e. disable csm, put your Ubuntu install medium back in (e.g. connect usb thumb drive) and boot from it. Then repartition the harddisk and install.

https://itsfoss.com/install-ubuntu/

Ok. There goes my day. I’m terrified. So many questions. I know it’s not your job to guide me through all that. But… does this mean all data will need to be moved/backed-up on another drive? Am I building the drive up from scratch? Re-installing ubuntu? Re-installing something else? Confused. :(

Yes. Backup data, then start from scratch. EFI needs a different partition schema.
The CSM of this mainboard doesn’t even provide enough resources for the mainboard to fully work in a 64bit OS. I guess it’s only provided to get some ancient 32bit Windows XP to install.

Thanks, @generix. Will do.

@generix and @aplattner, I did a fresh install. The attached log was collected right after the first reboot. dmesg.log (84.6 KB)

The board is still only providing 32bit resources. Please update bios and make sure above 4G decoding is really enabled.

Thanks. What are some of the key messages that indicate 64bit resources are/aren’t available? Ones like this?

[ 0.396029] pnp 00:05: disabling [mem 0xfed10000-0xfed17fff] because it overlaps 0000:03:00.0 BAR 1 [mem 0x00000000-0x3ffffffff 64bit pref]

It’s this:

[    0.333223] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.333224] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.333225] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.333226] pci_bus 0000:00: root bus resource [mem 0x90000000-0xdfffffff window]
[    0.333226] pci_bus 0000:00: root bus resource [mem 0xfc800000-0xfe7fffff window]
[    0.333227] pci_bus 0000:00: root bus resource [bus 00-fe]

Those are the memory windows that can be mapped to pcie devices. As you can see, all are just 32bit wide. Correct large/64bit BARs/Above 4G decoding should at least provide one window with 64bit width.

I enabled above 4G decoding (which was preventing bootup prior to my reinstall). New log file here. dmesg.log (77.1 KB)

This line implies more than 32bit, but doesn’t explicitly imply 64bit, yes?
pci_bus 0000:00: root bus resource [mem 0x4000000000-0x7fffffffff window]

Looks good now, 64bit resources enabled and Teslas are functional. Please install the driver now, should work.
Edit: on 64bit resource display, leading zeros are suppressed for readability. They just have to be longer than 32bits.

I installed CUDA and the driver and it appears that all is well. Thanks!!! Your responsiveness and expertise are MUCH appreciated! --Matt

Hi, I’m having the same problem on a DL580 G7, with the M40 24GB GPU, what can I do to avoid this error

Hi, I encounter the same problem on Gigabyte B550 Gaming X motherboard. CPU: AMD Ryzen 7 3700x. I plug the K80 into the PCIEX16 slot which is the integrated in CPU.

lspci | grep -i nvidia
03:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
03:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
0b:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
0c:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)

the dmesg log uploaded:
dmesg.log (87.9 KB)

The “above 4G decode” in BIOS was enabled. I found the problem is the same as above. This is new installed ubuntu 18.04. The same problem also on the new installed 20.04.

Is this a BIOS problem?