Ubuntu 20.04 Nvidia P100 - Recogized by OS, will not get used

We are using XCP-NG with PCI passthrough (fully supported) running Ubuntu 20.04 (fully up to date) with all updates installed for this version of Ubuntu. Latest Drivers installed using the software Updater in Ubuntu. Installed CUDA toolkit, nvida-smi will work to show that we are using 4MB out of 16GB. Nvidia X server settings will show up. Prime-Select nvidia has been done.
Whenever we run a GPU test, its only using the processors, no change in the print out of nvidia-smi… what are we doing wrong??