[EXPECTED BEHAVIOR] nvidia-smi reports incorrect memory usage in Windows 7/8 x64

[vacaloca]

The memory usage is correct in Ubuntu 13.10 x64, but not under Windows. If I enable TCC for the Quadro K6000, it reports correct memory usage for the K6000, but I can never get a correct readout for the Quadro K600 or the (GK208-based) GT 630 I use to drive my monitor.

I previously thought that the issue was related to running a GeForce card alongside a Quadro card, but the below proves otherwise.

Further, when I had the GT 630 installed by itself without the Quadro K6000, I see the same erroneous memory usage. GPU-z reports the correct usage, and the card will happily allocate memory until it reaches the respective maximum. This is a very strange bug… any ideas?

Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

C:\Windows\system32>nvidia-smi
Wed Nov 13 22:20:03 2013
+------------------------------------------------------+
| NVIDIA-SMI 331.65     Driver Version: 331.65         |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K600        WDDM  | 0000:02:00.0      On |                  N/A |
| 25%   43C    P8    N/A /  N/A |    986MiB /  1023MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Quadro K6000       WDDM  | 0000:03:00.0     Off |                  Off |
| 26%   30C    P8    19W / 225W |  12231MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0       600  Insufficient Permissions                           N/A      |
|    0       120  Insufficient Permissions                           N/A      |
|    0      3560  C:\Windows\Explorer.EXE                            N/A      |
|    0      1596  ...iles (x86)\Google\Chrome\Application\chrome.exe N/A      |
|    0      4324  ...iles (x86)\Google\Chrome\Application\chrome.exe N/A      |
|    1       600  Insufficient Permissions                           N/A      |
+-----------------------------------------------------------------------------+

[vacaloca]

Here’s the same output from Ubuntu 13.10 x64… same hardware:

MSI X79A-GD45 (8D), BIOS v12.4 (latest), UEFI/Secure Boot enabled

owner@Tesla:~/Desktop$ nvidia-smi
Wed Nov 13 22:27:09 2013       
+------------------------------------------------------+                       
| NVIDIA-SMI 331.20     Driver Version: 331.20         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K600         Off  | 0000:02:00.0      On |                  N/A |
| 25%   52C    P0    N/A /  N/A |    177MiB /  1020MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Quadro K6000        Off  | 0000:03:00.0     Off |                  Off |
| 26%   32C    P8    19W / 225W |     22MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|  No running compute processes found                                         |
+-----------------------------------------------------------------------------+

[vacaloca]

Just out of curiosity, I placed the Quadro K600 on a different system… this one is a Gigabyte X58A-UD3R (rev. 2.0) running an Intel(R) Core™ i7 CPU 930 @ 2.80GHz. Relatively clean install of WIndows 7 SP1 x64 and NVIDIA drivers, native BIOS (not UEFI boot like my previous system) and the same problem is apparent, so needless to say it’s not system dependent:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\Owner>nvidia-smi
Thu Nov 14 09:35:03 2013
+------------------------------------------------------+
| NVIDIA-SMI 331.65     Driver Version: 331.65         |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K600        WDDM  | 0000:03:00.0      On |                  N/A |
| 25%   37C    P8    N/A /  N/A |    984MiB /  1023MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0       512  Insufficient Permissions                           N/A      |
|    0      1812  C:\Windows\system32\Dwm.exe                        N/A      |
|    0      1252  ...6)\ATI Technologies\ATI.ACE\Core-Static\CCC.exe N/A      |
+-----------------------------------------------------------------------------+

[vacaloca]

Same system as above, and also happens with a card that doesn’t have an UEFI BIOS:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\Owner>nvidia-smi
Thu Nov 14 10:14:17 2013
+------------------------------------------------------+
| NVIDIA-SMI 331.65     Driver Version: 331.65         |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 430     WDDM  | 0000:03:00.0     N/A |                  N/A |
| 30%   32C  N/A     N/A /  N/A |    985MiB /  1023MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0            Not Supported                                               |
+-----------------------------------------------------------------------------+

[vacaloca]

For completeness, here are screenshots of the MSI X79A-GD45 (8D) motherboard system with previous driver versions showing the problem:

External Media

and this last one it is shown that GPUs in TCC mode report the correct memory usage:

External Media

[vacaloca]

Adding additional reproduction case with the Quadro K6000 by itself because NVIDIA didn’t want to acknowledge any oF the previous screenshots. Here is proof enough of the bug on the Gigabyte X58A-UD3R motherboard system. Quadro K6000 is fully supported by nvidia-smi, so this should be a valid repro case.

WDDM mode (incorrect memory usage shown):
External Media

TCC mode (correct memory usage shown):
External Media

[njuffa]

I would suggest filing a bug, using the bug reporting form linked from the registered developer website. Staring at the various screenshots, it is not clear to me which of the reported memory sizes are considered incorrect and why, so I would suggest pointing out the specific entries in question with their displayed and expected values in the bug report.

[vacaloca]

For example, in the last screenshot in WDDM mode, it shows memory usage of 12232 MiB, which is incorrect. The correct memory usage (22 MiB) is only reported when the card is in TCC mode. The Quadro K6000 was not driving a display at that point, nor running any code – nvidia-smi was called immediately after a reboot, and no programs are accessing the GPU.

I actually did file a bug through the registered developer website last week, and I received a reply from K. Kang mentioning the following:

@K. Kang

The “Used GPU Memory” is not available on Windows when running in WDDM mode. Because Windows KMD manages all the memory not NVIDIA driver.
So your observed behaviors on K6000 GPU are expected:
In TCC mode, the nvidia-smi.exe reports correct GPU memory usage;
In WDDM mode, the “GPU memory usage” is not available;

In nvidia-smi’s documentation, it’s described as following.

Used GPU Memory
Amount memory used on the device by the context. Not available on Windows when
running in WDDM mode because Windows KMD manages all the memory not NVIDIA
driver.

Needless to say, this issue is an existing limitation. I just hadn’t seen that blurb in the documentation. :oops: