It appears that versions of NVML included with NVIDIA GPU driver versions >= 500 have a bug whereby the NVML function to get the list of compute processes that are currently running on a given GPU device (
nvmlDeviceGetComputeRunningProcesses) returns compute processes across all GPU devices, rather than just the specified GPU device as was the case for previous versions of NVML included with 4xx series of NVIDIA GPU drivers (and as intended by the docs).
This issue has been observed to occur on Windows systems across a range of different GeForce/Quadro/Tesla GPU devices.
I have submitted a bug report here, but wanted to check whether other people had also experienced this or whether we now need to do something different when targeting systems with NVIDIA GPU driver version >= 500.
Reproducible sample code is available here: GitHub - peetw/nvml-bug