per-process resource accounting

I’d like to monitor per-process usage of the GPUs we’re using. I wrote a simple C++ program using OpenCV 3 which is using the OpenCL libraries for background subtraction in video. The tool decodes H264 video (on the CPU) and then runs background subtraction (in GPU). I’d like to run multiple such tasks concurrently (in separate processes) on the GPU and measure their use. I’m doing this on Windows on GeForce GTX 980Ti. I use “c:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe -l 1 -q -d ACCOUNTING” to monitor per-process usage.

When I start a single copy of the task, I see the following usage (and CPU at 25%).

Process ID                  : 10196
            GPU Utilization         : 25 %
            Memory Utilization      : 8 %
            Max memory usage        : 0 MiB
            Time                    : 15059 ms
            Is Running              : 0

This seems reasonable. However, when I start 2 concurrent copies of the same task, I see the following (CPU at 50%):

Process ID                  : 5636
           GPU Utilization         : 45 %
           Memory Utilization      : 16 %
           Max memory usage        : 0 MiB
           Time                    : 17689 ms
           Is Running              : 0
       Process ID                  : 8200
           GPU Utilization         : 43 %
           Memory Utilization      : 15 %
           Max memory usage        : 0 MiB
           Time                    : 17669 ms
           Is Running              : 0

When I start 4, I get this (CPU at 100%):

Process ID                  : 13592
            GPU Utilization         : 52 %
            Memory Utilization      : 19 %
            Max memory usage        : 0 MiB
            Time                    : 27207 ms
            Is Running              : 0
        Process ID                  : 14944
            GPU Utilization         : 54 %
            Memory Utilization      : 20 %
            Max memory usage        : 0 MiB
            Time                    : 27082 ms
            Is Running              : 0
        Process ID                  : 11492
            GPU Utilization         : 55 %
            Memory Utilization      : 20 %
            Max memory usage        : 0 MiB
            Time                    : 26987 ms
            Is Running              : 0
        Process ID                  : 7440
            GPU Utilization         : 55 %
            Memory Utilization      : 20 %
            Max memory usage        : 0 MiB
            Time                    : 26949 ms
            Is Running              : 0

It seems that nvidia-smi.exe is not really breaking the utilization by process; I’m running the same task so it’s average GPU utilization shouldn’t go up from 25% to 55%. And 4 * 55% > 100% so this doesn’t seem to make sense (because they were all running concurrently).

Am I missing something about this tool? I’d like to get per-process utilization numbers like I get for CPU using Windows resource monitor.

FYI, this is the info I get by running SMI by itself:

c:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe
Wed Oct 28 14:35:50 2015
+------------------------------------------------------+
| NVIDIA-SMI 358.50     Driver Version: 358.50         |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980 Ti WDDM  | 0000:05:00.0      On |                  N/A |
|  0%   43C    P8    17W / 250W |    953MiB /  6144MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0         4  C+G   Insufficient Permissions                     N/A      |
|    0       604  C+G   Insufficient Permissions                     N/A      |
|    0      5776  C+G   ...iles (x86)\Internet Explorer\IEXPLORE.EXE N/A      |
|    0      6556  C+G   Insufficient Permissions                     N/A      |
|    0      7068  C+G   ...icrosoft Office 15\root\office15\lync.exe N/A      |
|    0      7276  C+G   ...rosoft Office 15\root\office15\groove.exe N/A      |
|    0      9304  C+G   ...osoft Office 15\Root\Office15\WINWORD.EXE N/A      |
|    0     10608  C+G   ...osoft Office 15\root\office15\ONENOTE.EXE N/A      |
|    0     10620  C+G   ...iles (x86)\Internet Explorer\IEXPLORE.EXE N/A      |
|    0     10680  C+G   ...osoft Office 15\root\office15\OUTLOOK.EXE N/A      |
|    0     11432  C+G   C:\Program Files (x86)\VideoLAN\VLC\vlc.exe  N/A      |
|    0     12320  C+G   ...iles (x86)\Internet Explorer\IEXPLORE.EXE N/A      |
|    0     12980  C+G   ...ce 15\Root\Office15\NAMECONTROLSERVER.EXE N/A      |
|    0     13972  C+G   ...crosoft Office 15\root\office15\EXCEL.EXE N/A      |
+-----------------------------------------------------------------------------+

Thanks,
Peter