BAR1 Memory is 65GB on RTX A6000 - correct?

Virtual GPU Forums More vGPU Forums NVIDIA Virtual GPU Technology
1

Hello!

We are trying to get vGPU running with two A6000 in vSphere 7.0.3 Enterprise Plus and have completed installation. Now our VMs do not boot with the following error:

Could not initialize plugin ‘libnvidia-vgx.so’ for vGPU ‘nvidia_rtxa6000-12q’.

We’ve investigated all things, SR-IOV enabled, IOMMU enabled and switched the displaymode. Verifying the displaymode with nvidia-qmi -q shows this info for the two GPUs:

 BAR1 Memory Usage
        Total                             : 65536 MiB
        Used                              : 1 MiB
        Free                              : 65535 MiB

Is that indicating that the displaymode is set correctly for vGPU? It seems off, as the A6000 does not have 64GB of memory.

Greetings
Dominique

2

Mode is correct but I assume a VMWare parameter for the VM us still missing. How much sysmem is assigned?

3

The VM has 4GB of RAM and 12GB of VRAM (-12q). Should we give it more ram?

Also, we see that enabling SR-IOV does not succeed in ESXi for the GPUs, even after reboot the settings stays on “Needs reboot”. As this step is not mentioned in any documentation for vGPU, we are not sure if it is needed.

SR-IOV
SR-IOV1573×333 51.5 KB

PS: SR-IOV is enabled in BIOS, these are the settings:

BiosSettingBox
BiosSettingBox704×498 97.7 KB

4

We could fix the issue. Now we have received RTX 6000 Ada Generation and have a different problem - BAR1 shows 256MB for this card and we can’t change its display mode with the displaymodeselector tool. Is Bar1 of 256MB ok for using a RTX 6000 Ada for vGPU? If not, how do we change it?

@sschaber