cudaMemGetInfo returns similar result for 3 different GPUs

[sms.holding.3]

• OS: Windows 10
• CUDA : 11.5
• Language: C++

Hello there.
I have 3 similar 2080Ti GPUs on my PC. I need to query the amount of used memory of each GPU separately.
So I created this rudimentary application to test the CUDA functionalities

#include <iostream>
#include <cuda_runtime.h>

int main() {
int deviceCount;
cudaGetDeviceCount(&deviceCount);
std::cout << "Device count: " << deviceCount << std::endl;
for (int i = 0; i < deviceCount; i++) {
        cudaSetDevice(i);
        size_t free_byte;
        size_t total_byte;
        cudaMemGetInfo(&free_byte, &total_byte);
        size_t used_byte = total_byte - free_byte;

        // convert used_byte to MB
        auto used_mega_byte = used_byte / 1024 / 1024;
        auto free_mega_byte = free_byte / 1024 / 1024;
        auto total_mega_byte = total_byte / 1024 / 1024;
        std::cout << "Device " << i << " memory usage: " << free_mega_byte << " MB free, " << used_mega_byte << " 
        MB used, " << total_mega_byte << " MB total" << std::endl;
}

This is the output of the program on the console :

Device count: 3
Device 0 memory usage: 10119 MB free, 1144 MB used, 11263 MB total
Device 1 memory usage: 10119 MB free, 1144 MB used, 11263 MB total
Device 2 memory usage: 10119 MB free, 1144 MB used, 11263 MB total

This is the output of nvidia-smi :

Regarding the nvidia-smi output image, I would expect the output of the program to be similar to that of the
nvidia-smi command or at least one of them (the third one) should have given a different result.

It seems like cudaSetDevice is not working properly, as it gives the same result for all the cases.
I would be grateful if someone could clarify it to me what is happening and is there a workaround for this?
Is there any alternatives to query the amount of the used GPU memory reliably?
Thank you in advance.

[rs277]

Hopefully someone can give you a definitive answer. I found this older post in which Robert makes the following comment:
“Windows GPU memory for WDDM devices is managed by Windows, not by CUDA. the cudaMemGetInfo call will return information based on what the Windows GPU virtual memory manager (from Microsoft) tells it. It’s possible this may change from time to time, depending on what microsoft decides to do.”

Otherwise if you haven’t explored it already, nvidia-smi has various monitoring modes that can be targeted at specific metrics, memory being one, on multiple devices, which may do the job. I’m not sure if Windows has the man page with this info, that Linux offers.

I appreciate you may be seeking this data programatically, in which case the nvml library could be worth a look.

[sms.holding.3]

Thank you for your answer dear @rs277
Actually NVML is the solution of my issue and it gives more sensible results in which each GPU has different memory usage.
But there is another question, the amount of used memory for each GPU that is reported by NVML, is a little bit more than what I get from nvidia-smi.
You can see the the output of the program bellow:

GPU 0 memory usage: 8055 MB free, 3208 MB used, 11264 MB total
GPU 1 memory usage: 11041 MB free, 223 MB used, 11264 MB total
GPU 2 memory usage: 10270 MB free, 993 MB used, 11264 MB total

And the nvidia-smi output is as follows:

As you can see, there is subtle difference between the two reports, around 200 MB.
Where is this difference coming from and is it sensible?

[njuffa]

Did you consider the difference between reporting size in MB vs reporting it in MiB? Best I know nvidia-smi is based directly on NVML, so I would expect no differences to calling NVML directly. Well, maybe minor differences based on any memory that nvidia-smi and your app, respectively may use for themselves.

[sms.holding.3]

@sms.holding.3

3208

Oh, I just saw the amount of memory reported by nvidia-smi is in MiB, not MB. When I converted the values from MiB to MB, I almost get the same result.
Thank you so much.