Per-Device MPS controller fails due to the difference in device index order between what MPS recognizes and the index order used by other libraries

Accelerated Computing CUDA CUDA Programming and Performance
1

We have confirmed that on a GPU server with A4500 and A2000 GPUs, the device indices of the A2000 and A4500 differ from the device indices recognized by MPS.

For example, the current device order shown through nvidia-smi is as follows:

image
image636×207 5.86 KB

On the other hand, NVIDIA MPS recognizes the devices in the opposite order as follows:

image

This issue does not cause major problems in typical situations, but if the MPS Controller is specified with CUDA_VISIBLE_DEVICE as gpu_index, it causes a problem of creating an MPS server context for the wrong device. For example, when running the MPS controller with the command below, NVIDIA MPS creates a server context for the A4500. This is an incorrect result considering that the A2000 is the first device confirmed by nvidia-smi. As a result, the corresponding MPS client cannot utilize the GPU.

 CUDA_VISIBLE_DEVICES=0 nvidia-cuda-mps-control -f

image
image960×241 8.72 KB

image
image389×221 3.94 KB

This issue was confirmed with NVIDIA driver 535, and we verified whether this problem was resolved in the latest version 575, but it has not been fixed.

For this issue, a workaround is possible by specifying UUIDs instead of indices in CUDA_VISIBLE_DEVICES. However, considering that users can specify both indices and UUIDs in CUDA_VISIBLE_DEVICES, a fix seems necessary.

2

The nvidia-smi order is known to not necessarily map to the CUDA enumeration order. This is not a defect or bug in CUDA. I believe you can force CUDA to enumerate devices in the same order as nvidia-smi by including the CUDA_DEVICE_ORDER env var, properly set. Refer to the documentation.

3

First, thank you for your answer.
I didn’t know there was such a variable in CUDA environment variables. After testing, as you mentioned, nvidia-cuda-mps-control ran with the device order we wanted.

Actually, even if I knew about this environment variable, it would have been difficult to understand that it affects MPS… Thank you for your help!

However, I don’t quite understand why the default value of CUDA_DEVICE_ORDER is set to GPU performance order. This is because NVML, which we commonly use, sorts by PCI_BUS_ID regardless of GPU performance.
People who work at similar layers to us with Nvidia GPUs end up using both NVML and CUDA. In such cases, this difference in sorting order seems to create unnecessary complexity.

4

It’s been that way since the earliest days of CUDA to my knowledge. I could speculate as to reasons why, but the speculation wouldn’t change anything. It is the way it is, and its not likely to change, IMO.