Thanks for the quick response.
I had read your replies to:
forums.developer.nvidia.com/t/nvrm-this-pci-i-o-region-assigned-to-your-nvidia-device-is-invalid/81645
forums.developer.nvidia.com/t/installing-driver-fails-for-tesla-v100/83983
And looked for similar settings in my BIOS
https://www.supermicro.com/manuals/motherboard/5500/MNL-1076.pdf
Text search for “BAR” in manual comes back empty
Paragraph from: X8DTL-3/X8DTL-i/X8DTL-3F/X8DTL-iF User’s Manual
Page 4-15
PCI/PnP Configuration
SR-IOV Supported Single Root I/O Virtualization is an industry-standard mechanism that allows devices to be shared simultaneously among several virtual machines.
SR-IOV is capable of partitioning a PCI function into several virtual interfaces in order to share the resources of a PCI Express (PCIe) device under a virtual environment. The options are Disabled and Enabled.
Slot 1 PCI 33MHz/Slot 2 PCI 33MHz/Slot 3 PCIE x4 in X8 Slot/Slot 4 PCIE x8/ Slot 5 PCIE x4 in x8 Slot/Slot 6 PCIE x8 in x16 Slot This feature allows you to Enable or Disable any of the PCI slots indicated above. The options are Enable and Disable.
Appears to be a scant PCI section. (do you see something I’ve missed?)
As this was intended as a single vm passthrough install, SR-IOV was not enabled during host or VM install.
I have enabled it, and all other slots besides the x8 in x16 are disabled.
No change in error after reboot.
[ 7.242909] nvidia-nvlink: Nvlink Core is being initialized, major device number 242
[ 7.248827] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
[ 7.248827] NVRM: BAR1 is 0M @ 0x0 (PCI:0000:00:08.0)
[ 7.253912] NVRM: The system BIOS may have misconfigured your GPU.
[ 7.257084] nvidia: probe of 0000:00:08.0 failed with error -1
[ 7.260058] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 7.263182] NVRM: None of the NVIDIA devices were initialized.
[ 7.267954] nvidia-nvlink: Unregistered the Nvlink Core, major device number 242
On a side note: I tested trying to add the PCI hardware to another VM (creating) and it CentOS said no. So it looks to be exclusively assigned.
Addendum to VM config section of first post:
[root@vm1cent ~]# gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
I went with CentOS 7.6 off the cuda-installation-guide
Distribution Kernel*GCC GLIBC ICC PGI XLC CLANG
x86_64
CentOS 7.6 3.10 4.8.5 2.17 19.0 18.x, 19.x NO 8.0.0
Let me know if I can run any further hardware checks.
If you have suggestions on alternate OS configs (host or VM) I’m open.