Video Codec and Quadro P4000, P2000, P1000, P600, P400

Hello NVidia.

Is Video Codec SDK fully supported with all Quadro P* (without limitation of 2-encoder sessions per system) ?
How many encoder/decoder engines are present in chips GP106 (Quadro P2000) and GP107 (Quadro P1000/P600/P400) ?
What are Video Codec features and performance of new Quadro P* changed ?
Are all Quadro P* ready for passthrough in virtual enviroments/hypervisors (XenServer/KVM/vSphere/HyperV) ?
Please update https://developer.nvidia.com/video-encode-decode-gpu-support-matrix !

Thanks, Martin

refs:
http://www.nvidia.com/object/quadro-graphics-with-pascal.html
http://www.develop3d.com/hardware/quadro-p400-p600-p1000-p2000-p4000-cad-vr-viz-solidworks-vred-iray-pascal

Now fully updated: https://developer.nvidia.com/video-encode-decode-gpu-support-matrix
(but bugs in table P4000 - GP104 - 1x encoder ? Missing Tesla M10 !)

Still missing answer.

Quadro P2000 seems to does not support passthrough in virtual environment (Xen4.8, guest Fedora21+24, drivers 375.66,381.22). Quadro K2200 (maxwell gen1) (and others) works perfect in this environment.

Driver installation is ok, but fails to load driver when call “nvidia-smi” (as usual nvidia cryptic):
driver 381.22 - NVRM: RmInitAdapter failed! (0x24:0x65:1056)
driver 375.66 - NVRM: RmInitAdapter failed! (0x24:0x65:1060)
driver 378.13 - uncompilable on kernel >=4.10

“lspci -vv” - “Capabilities: [78] Express (v2): FLReset-” - Still unsupported basic requirement “function-level reset” for clean integration to virtualization environments, shame on NVidia !

Can NVidia decode, resolve and repair this error ?
Why did NVidia drop support for virtualization environments for cards >=X2000 ?
When will the virtualization environments be supported again ?
When will NVidia publish all requested information about limitation of its cards ?

I am waiting for answer for 5 days, if not answered returning card to distributor.
I am not expecting any relevant answer as usual. I found workaround.

Martin

I found stable workaround to make Quadro P2000 work in passthrough (nvidia-smi reports Virtualization mode : Pass-Through). Some relevant sample outputs for nvenc/nvdec (with nvenc application running):

# nvidia-smi
Tue May 16 19:53:28 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22                 Driver Version: 381.22                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P2000        Off  | 0000:00:05.0     Off |                  N/A |
| 64%   67C    P0    27W /  75W |    223MiB /  5053MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0       767    C   /root/thinvdi_encoder                          213MiB |
+-----------------------------------------------------------------------------+

# nvidia-smi -ac 3504,1721

# nvidia-smi -q
Timestamp                           : Tue May 16 19:51:49 2017
Driver Version                      : 381.22

Attached GPUs                       : 1
GPU 0000:00:05.0
    Product Name                    : Quadro P2000
    Product Brand                   : Quadro
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 1920
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : ....
    GPU UUID                        : GPU-....
    Minor Number                    : 0
    VBIOS Version                   : 86.06.3F.00.2E
    MultiGPU Board                  : No
    Board ID                        : 0x5
    GPU Part Number                 : 900-5G410-0300-000
    Inforom Version
        Image Version               : G410.0502.00.02
        OEM Object                  : 1.1
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : Pass-Through
    PCI
        Bus                         : 0x00
        Device                      : 0x05
        Domain                      : 0x0000
        Device Id                   : 0x1C3010DE
        Bus Id                      : 0000:00:05.0
        Sub System Id               : 0x11B3103C
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 3
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 32000 KB/s
        Rx Throughput               : 658000 KB/s
    Fan Speed                       : 64 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
        Sync Boost                  : Not Active
        Unknown                     : Not Active
    FB Memory Usage
        Total                       : 5053 MiB
        Used                        : 223 MiB
        Free                        : 4830 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 2 MiB
        Free                        : 254 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 4 %
        Memory                      : 1 %
        Encoder                     : 17 %
        Decoder                     : 0 %
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                Total               : N/A
        Aggregate
            Single Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                Total               : N/A
            Double Bit            
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 65 C
        GPU Shutdown Temp           : 104 C
        GPU Slowdown Temp           : 101 C
    Power Readings
        Power Management            : Supported
        Power Draw                  : 27.33 W
        Power Limit                 : 75.00 W
        Default Power Limit         : 75.00 W
        Enforced Power Limit        : 75.00 W
        Min Power Limit             : 75.00 W
        Max Power Limit             : 75.00 W
    Clocks
        Graphics                    : 1721 MHz
        SM                          : 1721 MHz
        Memory                      : 3499 MHz
        Video                       : 1544 MHz
    Applications Clocks
        Graphics                    : 1721 MHz
        Memory                      : 3504 MHz
    Default Applications Clocks
        Graphics                    : 1075 MHz
        Memory                      : 3504 MHz
    Max Clocks
        Graphics                    : 1721 MHz
        SM                          : 1721 MHz
        Memory                      : 3504 MHz
        Video                       : 1556 MHz
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes
        Process ID                  : 767
            Type                    : C
            Name                    : /root/thinvdi_encoder
            Used GPU Memory         : 213 MiB

# nvidia-smi pmon
# gpu     pid  type    sm   mem   enc   dec   command
# Idx       #   C/G     %     %     %     %   name
    0     767     C     3     0    16     0   thinvdi_encoder         
    0     767     C     3     0    16     0   thinvdi_encoder         

# nvidia-smi dmon
# gpu   pwr  temp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     %     %     %     %   MHz   MHz
    0    27    68     4     1    17     0  3499  1721
    0    27    68     4     1    17     0  3499  1721

Hi, the performance of th Quadro p2000 posted on the nvidia offical website is more different with the table shown up.The data is shown on https://developer.nvidia.com/nvidia-video-codec-sdk. Will you conform it?

Where exactly ?
Did you read “NVENC_Application_Note.pdf” ?

hi, mcerveny.
I don’t really understand that the stable workaround you are talking about.
Have you successfully make P2000 work in virtual environment(Xen4.8, guest Fedora21+24, drivers 375.66,381.22)?

Yes. I have my setup for virtualization stack (host - Fedora27/Xen4.10, guest - Fedora24/driver_381.22) with my (unpublished) workaround to make P2000 passthrough working (tested for CUDA/Encoder only). You should check your virtualization vendor and NVidia support if they have “professional”/“final” solution after 7 month.
Check passthrough compatibility lists:

thanks. you are very professional.

It’s over a year later, myself and the internet would greatly appreciate the workaround used to pass the P2000 card through successfully. I’ll sign an NDA or whatever you want to do not share it. I’m running esxi 6.7 trying to pass it through to Ubuntu 18.04 LTS VM. Thanks for taking the time.

-Sully

It should be officially supported in ESXi 6.5 from U1/U2 - check https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vdga&details=1&gpuDeviceModels=NVIDIA%20Quadro%20P2000 and contact official support.

My old workaround may work in “Citrix Hypervisor” (rebranded XenServer) but probably not in ESXi because it needs some startup scripting on host (I am not aware of “crippled linux” in ESXi).

I do not have problem to publish workaround but I tested it only with my setup and for XEN + CUDA/nvENC usage and I do not need old workaround with newer guest drivers. Newer drivers have “new feature” that allows use HW only “one time” after host reboot. Restarting guest or any nvidia-use-program lead to error in guest driver. The first nvidia-use-program in guest must be “nvidia-smi -pm 1” (new workaround) (tested with P2000 passthrough from host XEN 4.10.1 to guest 4.16.14-200.fc27.x86_64, Nvidia driver 396.26).

Old workaround was easy. I found that my P2000 must be initialized by Nvidia host old driver first and than it is possible to initialize by Nvidia guest old driver without error.

  • The device should not be blacklisted in startup (eg. not "xen-pciback.hide=(XX:00.0)" in startup).
  • There must be installed supported "vGPU NVidia" driver for hypervisor (eg. http://www.nvidia.com/object/virtual-gpu-software-driver.html).
  • It should be checked by "nvidia-smi".
  • Then remove from host to be available to guest:
    nvidia-smi
    echo 0000:XX:00.0 > /sys/bus/pci/devices/0000\:XX\:00.0/driver/unbind 
    echo 0000:XX:00.0 > /sys/bus/pci/drivers/pciback/new_slot
    echo 0000:XX:00.0 > /sys/bus/pci/drivers/pciback/bind
    
  • It (probably) should be reinitialized after every guest shutdown:
    echo 0000:XX:00.0 > /sys/bus/pci/drivers/pciback/unbind 
    echo 0000:XX:00.0 > /sys/bus/pci/drivers/nvidia/bind 
    nvidia-smi
    echo 0000:XX:00.0 > /sys/bus/pci/devices/0000\:XX\:00.0/driver/unbind 
    echo 0000:XX:00.0 > /sys/bus/pci/drivers/pciback/new_slot
    echo 0000:XX:00.0 > /sys/bus/pci/drivers/pciback/bind