vGPU on GTX or RTX or Quadro

NVIDIA Virtual GPU Forums Virtual GPU NVIDIA Virtual GPU Technology
#1

My opinion: NVidia and commercial hypervisor vendors unbelievable escalated the cost of vGPU technology and NVidia support is unusable for me. Ok, this approach forced me to accept challenge to resolve this problem with own simplified (NVidia unsupported) Xen based virtualization stack. Few days of experiments (to make it compatible with GTX/Quadro) and now vGPU can be run with any Xen (few public XenServer patches and some more for vgpu ioreq_server (Citrix stops to distribute vgpu sources from XenServer 7.5)), any compatible NVidia GTX or Quadro (small 42 lines "magic" script daemon and change one line in driver install script but no need to patch any NVIDIA files), any linux kernel and any linux distribution. I will not publish how - challenge yourself.
I used my notebook (i7-4710HQ/C220 VT-d aware with Intel graphics (as display frontend) and NVidia GTX860M (as virtualization backend)) as demonstration of Xen with vGPU virtualization server (Dom0 - xen4.10.1 with Fedora 28 and kernel 4.16, Windows DomU - Win2008r2 with enabled CUDA and running GPU-Z, CUDA-Z, Unigine Haeven, Linux DomU - CentOS7.5 with enabled CUDA and running nvenc Video Codec SDK sample):


UPDATES:
Verified with Grid 4.6 (license less), 6.1 …
Verified with Grid/vGPU SW 6.2 - xen 4.11, dom0 - Fedora 28/kernel 4.16 (kernels 4.17-4.19 do not work with xen)
Verified with vGPU SW 7.1 - xen 4.11, dom0 - Fedora 28/kernel 4.16/4.20, xen 4.11.1 - Fedora 29/kernel 4.20 (with Quadro K2200)
Verified with vGPU SW 8.0 - xen 4.12, dom0 - Fedora 29/kernel 5.0 (with Quadro K2200)
Verified with vGPU SW 9.0 - xen 4.12, dom0 - Fedora 29/kernel 5.0 (with Quadro K2200)

#2

Some happy announcements:

I bought cheap GTX1080 card today ($460) to test vGPU virtualization with TeslaP4 profiles and I was successful. There is performance advantage - TeslaP4 has power limit 75W (base clock ~860MHz) and my GTX1080 has power limit 180W (base clock ~1632MHz (boosted in Superposition benchmark to ~1873MHz (capped by 180W limit) with frame_rate_limiter=0). Conclusion: GTX1080 performance ± equals to TeslaT4 and NVENC is 2x better (raw FPS) or 5x better (FPS recomputed with delivered VDI) (see notes) than TeslaT4. TeslaT4 NVENC is unusable for VDI.

I bought GTX980 card from EBay ($160) to test vGPU virtualization with TeslaM60 profiles and I was also successful. There is performance advantage - TeslaM60 has power limit 300W (2xGM204 chips and 2x8GB RAM) (base clock ~899MHz) and my GTX980 has power limit 180W (1xGM204 and 1x4GB RAM) (base clock ~1152MHz (boosted in Superposition benchmark to ~1265MHz (capped by 180W limit) with frame_rate_limiter=0).

Notes:

  • screenshots from virtualized "win1 (win2008r2)" with putty to virtualized "linux1 (centos75)" (with license server) and to XEN domain0 (srv)
  • available average FPS for one VDI instance (H.264, Low latency High Performance single pass, reference NVENC speeds are taken from NVidia Video Codec SDK 9.0 (NVENC_Application_Note.pdf), GPU clocks from wikipedia) [img]https://i.postimg.cc/dQrrJDb0/nvenc-vgpu-comparison.jpg[/img]

UPDATES:
verified with vGPU SW 6.4 + XEN 4.12 + Fedora 29/kernel 5.0 (dom0) + GTX1080
verified with vGPU SW 8.0 - XEN 4.12 + Fedora 29/kernel 5.0 (dom0) + GTX1080
verified with vGPU SW 9.0 - XEN 4.12 + Fedora 29/kernel 5.0 (dom0) + GTX1080
updated NVENC performance per vGPU to vGPU SW 8.0 (RTX 6000/8000, profile *-1B4) - RTX 6000/8000 do not have "A" or "B" profiles (pay more). There are limit to 32 profiles per GPU. RTX8000-1Q with 4k and NVENC guarantees fantastic 5 FPS and 16GB vRAM are lost - best profile ever ! I do not believe that user VDI statistical multiplexing is usable without UX impact with less than 15 FPS in average especially in Windows10 with many animated effects running everywhere (and "running video", "running google maps", "running powerpoint transition effects" as NVidia marketing presentation asserts). I think that Turing generation is absolutely unsuitable for vGPU based VDI.
nvenc_vgpu_comparison.ods.gz (11.2 KB)

#3

FYI:

NVidia and/or hypervisor vendors do not allow mixed vGPU types/profiles on one GPU chip (https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#homogeneous-grid-vgpus) and neither it will be possible in the next vGPU SW 8.0 release. It does not make any sense not to allow this from my engineering point of view because it only differs in allocation of static vRAM memory. The “best effort” scheduler works equally in all vGPU types/profiles (based on cooperative multitasking with frame rate limit as task switch trigger).
I added two lines to my magic script that expands vRAM memory allocation (framebufferlength+reserved_fb). Practically any size should work (probably rounded to 64MB). Maximum resolution and number of heads in “P4-1Q” can be changed accordingly. You do not need additional Windows reboot with “vRAM memory size change” (it was needed with profile change due to PCIid change) - useful with disk clones. I successfully tested mixed vGPU types/profiles on one GPU chip - srv is XEN virtualization DOM0 (XEN4.12.0+linux5.0), “P4-1Q with 2GB vRAM” for windows (following screenshot is from this windows) (GPU-Z+CUDA-Z+Heaven+Superposition(“medium” needs 1.3GB vRAM) (performance is limited by FRL)), P4-1Q for linux (centos75 with stream encoder) and “P4-1Q with 4GB vRAM” for linux (linux10 with Xorg+Heaven):

PS: Compare TeslaT4 FP32(single)=8.141 TFLOPS, FP64(double)=254.4 GFLOPS with virtualized GTX1080 (without other benchmark running, GTX1080 lacks Tensor and Ray-tracing cores):

UPDATES:
Verified with vGPU SW 8.0 - xen 4.12, dom0 - Fedora 29/kernel 5.0 (with GTX1080)
Verified with vGPU SW 9.0 - xen 4.12, dom0 - Fedora 29/kernel 5.0 (with GTX1080)

#4

Some happy announcement:

I bought cheapest RTX 2070 Super card today to test vGPU virtualization with TeslaT4 profiles and I was successful. There is performance advantage - TeslaT4 has power limit 70W (base clock ~585MHz) and my RTX2070Super has power limit 215W (base clock ~1605MHz (boosted in Superposition benchmark to ~1890MHz (capped by 215W limit) with frame_rate_limiter=0). I successfully tested mixed vGPU types/profiles on one GPU chip. CUDA-Z shows improvements in integer benchmarks, memory throughput but same performance in floating point as GTX1080 (same number of SM and SM frequency) and higher power consumption than GTX1080. Overview screenshot from virtualized windows with putty to XEN Dom0 (XEN 4.12.1, Fedora30, vGPU 8.1):

Let’s compare “GeForce RTX2070Super” and “Quadro RTX 8000” in 4k Superposition vGPU benchmark:

card           RTX2070S   RTX8000    ratio
4k result      7874       9686       <b>+23%</b>
MEM bandwidth  448 GB/s   672 GB/s   +50%
MEM size       8 GB       48 GB      +500%
CORE speed     2560*1605  4608*1395  +56%
price          $499       $5500      <b>+1000%</b>

There are 4k Superposition benchmark results without virtualization: 8254 vs. 11797 (ratio +43%) (see https://www.servethehome.com/nvidia-quadro-rtx-8000-gpu-review/5/).

UPDATES:
Verified with vGPU SW 9.1 (430.46/430.46/431.79) - xen 4.12.1, dom0 - Fedora 30/kernel 5.0.9 (with RTX2070Super)
Verified with vGPU SW 9.2 (430.67/432.08) - xen 4.12.1, dom0 - Fedora 30/kernel 5.3.15-200 (with RTX2070Super)
Verified with vGPU SW 10.0 (440.43/441.66) - xen 4.13.0, dom0 - Fedora 31/kernel 5.3.16-300 (with RTX2070Super)
Verified with vGPU SW 10.1 (440.53/442.06) - xen 4.13.0, dom0 - Fedora 31/kernel 5.3.16-300 (with RTX2070Super) NOW with HDR support !
Verified with vGPU SW 10.2 (440.87/443.05) - xen 4.13.0, dom0 - Fedora 31/kernel 5.6.8-200 (with RTX2070Super)
Verified with vGPU SW 11.0 (450.55/451.48) - xen 4.13.0, dom0 - Fedora 31/kernel 5.6.8-200 (with RTX2070Super) (451.48_grid_server2012R2_64bit_international.exe - Win7/8/20008r2 are working with some signing errors)
Verified with vGPU SW 11.1 (450.80/452.39) - xen 4.13.0, dom0 - Fedora 32/kernel 5.8.13-200 (with RTX2070Super) (There is performance lost ~5% because NVIDIA lower performance/boost throttling point from 80°C to 75°C and also usual ~10% performance penalty if you are using Win10 (versus Win7).)
Verified with vGPU SW 11.2 (450.89/452.57) - xen 4.13.0, dom0 - Fedora 32/kernel 5.8.13-200 (with RTX2070Super) (There is additional performance lost ~4% with this release! So summarized: One year old Win7/vgpu8.1 have about 18.9% better performance than Win10/vgpu11.2.)

#5

FYI:
NVIDIA use the same technology (Xen HVM and GeForce Cards) for RTX SERVER CLOUD GAMING:

  • GeForce RTX T10 (16GB, TU102)
    • passthrough
    • vGPU profiles - GRID RTX T10-16, GRID RTX T10-8, GRID RTX T10-4, GRID RTX T10-1, GRID RTX T10-0, GeForce RTX T10-16, GeForce RTX T10-8, GeForce RTX T10-4
    • license - GRID vGaming (GRID-vGaming,8.0)
  • GeForce RTX T10x (8GB, Turing)
    • vGPU profiles - GRID RTX T10x-8, GRID RTX T10x-4, GRID RTX T10x-2, GeForce RTX T10x-8, GeForce RTX T10x-4, GeForce RTX T10x-2

Newcomers:

  • NVIDIA RTX A6000 (48GB, GA102)
  • NVIDIA A40 (48GB, GA102)
  • NVIDIA A10 (24GB, Ampere)
    • with commercial vGPU profiles - NVIDIA A10-1B, NVIDIA A10-2B, NVIDIA A10-1Q, NVIDIA A10-2Q, NVIDIA A10-3Q, NVIDIA A10-4Q, NVIDIA A10-6Q, NVIDIA A10-8Q, NVIDIA A10-12Q, NVIDIA A10-24Q, NVIDIA A10-6, NVIDIA A10-8, NVIDIA A10-12, NVIDIA A10-24, NVIDIA A10-4C, NVIDIA A10-6C, NVIDIA A10-8C, NVIDIA A10-12C, NVIDIA A10-24C